[jira] [Created] (AURORA-1884) Aurora observer does not get updated when mesos preempts tasks
Kunal Thakar created AURORA-1884: Summary: Aurora observer does not get updated when mesos preempts tasks Key: AURORA-1884 URL: https://issues.apache.org/jira/browse/AURORA-1884 Project: Aurora Issue Type: Story Components: Observer Reporter: Kunal Thakar The 'active tasks' table in the observer UI does not reflect the real status of those tasks. In particular, when the mesos QoS killer (http://mesos.apache.org/documentation/latest/oversubscription/) preempts an Aurora task, it still shows up in the UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1734) Configurable Metadata prefix
[
https://issues.apache.org/jira/browse/AURORA-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379836#comment-15379836
]
Kunal Thakar commented on AURORA-1734:
--
What is the purpose of the metadata prefix on the mesos labels? Will there ever
be a case of label collision in an individual task?
> Configurable Metadata prefix
>
>
> Key: AURORA-1734
> URL: https://issues.apache.org/jira/browse/AURORA-1734
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
>Reporter: Renan DelValle
>Assignee: Renan DelValle
>Priority: Trivial
>
> Currently, a prefix ("org.apache.aurora.metadata.") is injected into the
> metadata key in the scheduler. It would be beneficial to allow users to set
> their own metadata prefix (including an empty string).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AURORA-1643) Support authentication between announcer and ZK
[ https://issues.apache.org/jira/browse/AURORA-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206793#comment-15206793 ] Kunal Thakar commented on AURORA-1643: -- Up for review https://reviews.apache.org/r/45042/ > Support authentication between announcer and ZK > --- > > Key: AURORA-1643 > URL: https://issues.apache.org/jira/browse/AURORA-1643 > Project: Aurora > Issue Type: Story >Reporter: Kunal Thakar > > We want to restrict access to the ZK service discovery cluster through ACLs. > Currently, the announcer does not support creating ZK nodes with ACLs. The > Kazoo client supports ACLs, so it should be straightforward to plumb in > support for ACLs in the announcer (how do we pass ACL credentials to the > announcer is another question). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1643) Support authentication between announcer and ZK
[ https://issues.apache.org/jira/browse/AURORA-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197772#comment-15197772 ] Kunal Thakar commented on AURORA-1643: -- Proposal: - The Kazoo client, which is instantiated in the announcer supports 'default_acl' (http://kazoo.readthedocs.org/en/latest/api/client.html#kazoo.client.KazooClient), which will be used for subsequent node creations. - The 'default_acl' can be generated by calling make_digest_acl (http://kazoo.readthedocs.org/en/latest/api/security.html#kazoo.security.make_digest_acl). The username, password and perms can be made available to the announcer through a new config file or through environment variables. > Support authentication between announcer and ZK > --- > > Key: AURORA-1643 > URL: https://issues.apache.org/jira/browse/AURORA-1643 > Project: Aurora > Issue Type: Story >Reporter: Kunal Thakar > > We want to restrict access to the ZK service discovery cluster through ACLs. > Currently, the announcer does not support creating ZK nodes with ACLs. The > Kazoo client supports ACLs, so it should be straightforward to plumb in > support for ACLs in the announcer (how do we pass ACL credentials to the > announcer is another question). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1643) Support authentication between announcer and ZK
[
https://issues.apache.org/jira/browse/AURORA-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15200086#comment-15200086
]
Kunal Thakar commented on AURORA-1643:
--
For our usecase, we want to set all but Admin perms on the nodes (Create, Read,
Delete, Write). The file format should support perms as well in my opinion. How
about the following format?
{
'username': ,
'password': ,
'permissions': {
'read': ,
'write': ,
'create': ,
'delete': ,
'admin':
}
}
> Support authentication between announcer and ZK
> ---
>
> Key: AURORA-1643
> URL: https://issues.apache.org/jira/browse/AURORA-1643
> Project: Aurora
> Issue Type: Story
>Reporter: Kunal Thakar
>
> We want to restrict access to the ZK service discovery cluster through ACLs.
> Currently, the announcer does not support creating ZK nodes with ACLs. The
> Kazoo client supports ACLs, so it should be straightforward to plumb in
> support for ACLs in the announcer (how do we pass ACL credentials to the
> announcer is another question).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AURORA-1643) Support authentication between announcer and ZK
Kunal Thakar created AURORA-1643: Summary: Support authentication between announcer and ZK Key: AURORA-1643 URL: https://issues.apache.org/jira/browse/AURORA-1643 Project: Aurora Issue Type: Story Reporter: Kunal Thakar We want to restrict access to the ZK service discovery cluster through ACLs. Currently, the announcer does not support creating ZK nodes with ACLs. The Kazoo client supports ACLs, so it should be straightforward to plumb in support for ACLs in the announcer (how do we pass ACL credentials to the announcer is another question). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1611) Add support for IP addresses instead of just hostnames in Announcer
[ https://issues.apache.org/jira/browse/AURORA-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15190237#comment-15190237 ] Kunal Thakar commented on AURORA-1611: -- Sorry if I wasn't clear. We want to start mesos slaves with --hostname=, which means that the announcer has access to only the hostname. In this case, in order to announce the IP instead of the hostname, we can either run an external command (the way I am proposing) or do a DNS resolution inside the announcer. Running an external command is more flexible, which is why I prefer that approach. Another alternative is to change the way AssignedTask is generated (https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java#L112) and insert an IP in that struct, which will then be accessible to the announcer. I will add some tests to my proposed change and send it out for review barring any objections. Thanks! > Add support for IP addresses instead of just hostnames in Announcer > --- > > Key: AURORA-1611 > URL: https://issues.apache.org/jira/browse/AURORA-1611 > Project: Aurora > Issue Type: Story >Reporter: Kunal Thakar > > The thermos announcer is passing hostnames in the serviceEndpoint object that > is sent to ZK. We want the announcer to send IPs instead or in addition to > the hostname that is passed along. What is the best way to achieve this? I am > thinking of adding a new attribute to the Endpoint struct > (https://github.com/apache/aurora/blob/master/commons/src/main/thrift/org/apache/aurora/common/thrift/endpoint.thrift). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1611) Add support for IP addresses instead of just hostnames in Announcer
[ https://issues.apache.org/jira/browse/AURORA-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189922#comment-15189922 ] Kunal Thakar commented on AURORA-1611: -- Any updates? Is this approach acceptable? > Add support for IP addresses instead of just hostnames in Announcer > --- > > Key: AURORA-1611 > URL: https://issues.apache.org/jira/browse/AURORA-1611 > Project: Aurora > Issue Type: Story >Reporter: Kunal Thakar > > The thermos announcer is passing hostnames in the serviceEndpoint object that > is sent to ZK. We want the announcer to send IPs instead or in addition to > the hostname that is passed along. What is the best way to achieve this? I am > thinking of adding a new attribute to the Endpoint struct > (https://github.com/apache/aurora/blob/master/commons/src/main/thrift/org/apache/aurora/common/thrift/endpoint.thrift). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1611) Add support for IP addresses instead of just hostnames in Announcer
[ https://issues.apache.org/jira/browse/AURORA-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186044#comment-15186044 ] Kunal Thakar commented on AURORA-1611: -- It won't. We are not planning to deploy aurora client on all engineer laptops, which is the location of the problem. Also, as I mentioned above, we are planning to run other frameworks in the mesos cluster. Additionally, this sort of approach will also be required to support the IP-per-container world (https://mesosphere.com/blog/2015/12/02/ip-per-container-mesos/), which is not very far in the future. > Add support for IP addresses instead of just hostnames in Announcer > --- > > Key: AURORA-1611 > URL: https://issues.apache.org/jira/browse/AURORA-1611 > Project: Aurora > Issue Type: Story >Reporter: Kunal Thakar > > The thermos announcer is passing hostnames in the serviceEndpoint object that > is sent to ZK. We want the announcer to send IPs instead or in addition to > the hostname that is passed along. What is the best way to achieve this? I am > thinking of adding a new attribute to the Endpoint struct > (https://github.com/apache/aurora/blob/master/commons/src/main/thrift/org/apache/aurora/common/thrift/endpoint.thrift). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1611) Add support for IP addresses instead of just hostnames in Announcer
[ https://issues.apache.org/jira/browse/AURORA-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186027#comment-15186027 ] Kunal Thakar commented on AURORA-1611: -- The service discovery usecase is fine. Human operators interacting with this mesos cluster where the agents are started with --hostname= have to jump through multiple hoops to ssh to the machines, etc. Keep in mind that multiple frameworks will share the cluster, so it seems onerous to configure mesos this way to support this usecase. > Add support for IP addresses instead of just hostnames in Announcer > --- > > Key: AURORA-1611 > URL: https://issues.apache.org/jira/browse/AURORA-1611 > Project: Aurora > Issue Type: Story >Reporter: Kunal Thakar > > The thermos announcer is passing hostnames in the serviceEndpoint object that > is sent to ZK. We want the announcer to send IPs instead or in addition to > the hostname that is passed along. What is the best way to achieve this? I am > thinking of adding a new attribute to the Endpoint struct > (https://github.com/apache/aurora/blob/master/commons/src/main/thrift/org/apache/aurora/common/thrift/endpoint.thrift). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (AURORA-1611) Add support for IP addresses instead of just hostnames in Announcer
[ https://issues.apache.org/jira/browse/AURORA-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Thakar reopened AURORA-1611: -- While this approach works, it is causing a big usability issue for us. The mesos and aurora UIs now show private IPs instead of hostnames, and we are not able to easily ssh to those machines with the datacenter private IPs (without resorting to a reverse lookup). I am proposing this small change: https://reviews.apache.org/r/44532 In this approach, executors will have to be started with a parameter 'announcer-set-hostname-cmd'. For example, --announcer-set-hostname-cmd = 'cat /etc/mesos-slave/ip' > Add support for IP addresses instead of just hostnames in Announcer > --- > > Key: AURORA-1611 > URL: https://issues.apache.org/jira/browse/AURORA-1611 > Project: Aurora > Issue Type: Story >Reporter: Kunal Thakar > > The thermos announcer is passing hostnames in the serviceEndpoint object that > is sent to ZK. We want the announcer to send IPs instead or in addition to > the hostname that is passed along. What is the best way to achieve this? I am > thinking of adding a new attribute to the Endpoint struct > (https://github.com/apache/aurora/blob/master/commons/src/main/thrift/org/apache/aurora/common/thrift/endpoint.thrift). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1611) Add support for IP addresses instead of just hostnames in Announcer
[ https://issues.apache.org/jira/browse/AURORA-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139231#comment-15139231 ] Kunal Thakar commented on AURORA-1611: -- Thanks, I tested it with the workaround and it is fine. Not an ideal solution, but it works! Feel free to resolve this issue. > Add support for IP addresses instead of just hostnames in Announcer > --- > > Key: AURORA-1611 > URL: https://issues.apache.org/jira/browse/AURORA-1611 > Project: Aurora > Issue Type: Story >Reporter: Kunal Thakar > > The thermos announcer is passing hostnames in the serviceEndpoint object that > is sent to ZK. We want the announcer to send IPs instead or in addition to > the hostname that is passed along. What is the best way to achieve this? I am > thinking of adding a new attribute to the Endpoint struct > (https://github.com/apache/aurora/blob/master/commons/src/main/thrift/org/apache/aurora/common/thrift/endpoint.thrift). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1611) Add support for IP addresses instead of just hostnames in Announcer
Kunal Thakar created AURORA-1611: Summary: Add support for IP addresses instead of just hostnames in Announcer Key: AURORA-1611 URL: https://issues.apache.org/jira/browse/AURORA-1611 Project: Aurora Issue Type: Story Reporter: Kunal Thakar The thermos announcer is passing hostnames in the serviceEndpoint object that is sent to ZK. We want the announcer to send IPs instead or in addition to the hostname that is passed along. What is the best way to achieve this? I am thinking of adding a new attribute to the Endpoint struct (https://github.com/apache/aurora/blob/master/commons/src/main/thrift/org/apache/aurora/common/thrift/endpoint.thrift). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1569) Support custom serverset path for announcements
[ https://issues.apache.org/jira/browse/AURORA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086264#comment-15086264 ] Kunal Thakar commented on AURORA-1569: -- I agree, that was the path I was going down initially. But that solution is more involved (requires adding the ability to specify a different AnnouncerCheckerProvider implementation and also a rudimentary template renderer), which is why I chose this solution. > Support custom serverset path for announcements > --- > > Key: AURORA-1569 > URL: https://issues.apache.org/jira/browse/AURORA-1569 > Project: Aurora > Issue Type: Story > Components: Executor >Reporter: Kunal Thakar >Assignee: Kunal Thakar > > The Announcer serverset path is not flexible enough. You can only change the > root of the path (/$ROOT/role/environment/jobname). As a user with different > service discovery schema, this doesn't fit our needs. We want to be able to > specify the serverset announce path with the job spec. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1569) Support custom serverset path for announcements
[ https://issues.apache.org/jira/browse/AURORA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15086064#comment-15086064 ] Kunal Thakar commented on AURORA-1569: -- Related code review https://reviews.apache.org/r/41809/ > Support custom serverset path for announcements > --- > > Key: AURORA-1569 > URL: https://issues.apache.org/jira/browse/AURORA-1569 > Project: Aurora > Issue Type: Story > Components: Executor >Reporter: Kunal Thakar >Assignee: Kunal Thakar > > The Announcer serverset path is not flexible enough. You can only change the > root of the path (/$ROOT/role/environment/jobname). As a user with different > service discovery schema, this doesn't fit our needs. We want to be able to > specify the serverset announce path with the job spec. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1569) Support custom serverset path for announcements
[ https://issues.apache.org/jira/browse/AURORA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083688#comment-15083688 ] Kunal Thakar commented on AURORA-1569: -- Something like . > Support custom serverset path for announcements > --- > > Key: AURORA-1569 > URL: https://issues.apache.org/jira/browse/AURORA-1569 > Project: Aurora > Issue Type: Story > Components: Executor >Reporter: Kunal Thakar > > The Announcer serverset path is not flexible enough. You can only change the > root of the path (/$ROOT/role/environment/jobname). As a user with different > service discovery schema, this doesn't fit our needs. We want to be able to > specify the serverset announce path with the job spec. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1569) Support custom serverset path for announcements
[ https://issues.apache.org/jira/browse/AURORA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15081837#comment-15081837 ] Kunal Thakar commented on AURORA-1569: -- Ok. I have updated the code review with the above recommendation. > Support custom serverset path for announcements > --- > > Key: AURORA-1569 > URL: https://issues.apache.org/jira/browse/AURORA-1569 > Project: Aurora > Issue Type: Story > Components: Executor >Reporter: Kunal Thakar > > The Announcer serverset path is not flexible enough. You can only change the > root of the path (/$ROOT/role/environment/jobname). As a user with different > service discovery schema, this doesn't fit our needs. We want to be able to > specify the serverset announce path with the job spec. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1569) Support custom serverset path for announcements
[ https://issues.apache.org/jira/browse/AURORA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075548#comment-15075548 ] Kunal Thakar commented on AURORA-1569: -- These are custom attributes that can be passed through TaskConfig only as metadata. > Support custom serverset path for announcements > --- > > Key: AURORA-1569 > URL: https://issues.apache.org/jira/browse/AURORA-1569 > Project: Aurora > Issue Type: Story > Components: Executor >Reporter: Kunal Thakar > > The Announcer serverset path is not flexible enough. You can only change the > root of the path (/$ROOT/role/environment/jobname). As a user with different > service discovery schema, this doesn't fit our needs. We want to be able to > specify the serverset announce path with the job spec. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AURORA-1569) Support custom serverset path for announcements
[ https://issues.apache.org/jira/browse/AURORA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15075545#comment-15075545 ] Kunal Thakar commented on AURORA-1569: -- We want a format that conforms to the way services are named in our existing toolchain (deploy system, service discovery system, etc). If we use the aurora scheme, we will have to cram our naming scheme into the 'jobname' field using some arbitrary delimiters, which makes the whole thing brittle (and we also lose the hierarchical structure of ZK). > Support custom serverset path for announcements > --- > > Key: AURORA-1569 > URL: https://issues.apache.org/jira/browse/AURORA-1569 > Project: Aurora > Issue Type: Story > Components: Executor >Reporter: Kunal Thakar > > The Announcer serverset path is not flexible enough. You can only change the > root of the path (/$ROOT/role/environment/jobname). As a user with different > service discovery schema, this doesn't fit our needs. We want to be able to > specify the serverset announce path with the job spec. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AURORA-1569) Support custom serverset path for announcements
Kunal Thakar created AURORA-1569: Summary: Support custom serverset path for announcements Key: AURORA-1569 URL: https://issues.apache.org/jira/browse/AURORA-1569 Project: Aurora Issue Type: Story Components: Executor Reporter: Kunal Thakar The Announcer serverset path is not flexible enough. You can only change the root of the path (/$ROOT/role/environment/jobname). As a user with different service discovery schema, this doesn't fit our needs. We want to be able to specify the serverset announce path with the job spec. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
