[OpenStack-Infra] Zuul info for PTG

[James E. Blair]

Hi!

We're meeting in person at the Project Team Gathering in Atlanta soon,
so I wanted to share the current state of Zuul and how we're thinking
about setting up jobs so we're all able to contribute to our goal of
actually running Zuul v3 when we are there.

Also, we might borrow some of these words for the user manual later.

There's a lot of info in the spec about how jobs are defined in Zuul
v3, some of it general, and some of it directed at how we will do things
specifically for OpenStack.  But there are also areas where it is
intentionally vague as it was not clear at the time the best way to
actually implement some ideas.  We're much closer now, and can fill in
some of the gaps.  I'd like to do that and present this information in
the context of how I expect us to use it in the OpenStack project.

How Configuration Works
---

The first significant change is that Zuul's configuration is contained
almost entirely in git repositories that Zuul directly manages.  We will
have a small snippet of YAML that will live in project-config which our
system-config driven ansible-puppet system will install on the main Zuul
server.  That YAML file is the tenant config file for Zuul, which lists
the tenants we will have (we will probably start with just one,
"OpenStack", but we might further divide later), and the repos that are
associated with that tenant (pretty much everything in Gerrit).
Eventually we may have Zuul automatically query these, but for now, we
will have to list them.  However, unlike our current 19,000 line
zuul.yaml file, all we need to do is list the names of the repos.  So
it'll only be 1,700 lines.  :)

For the PTG, we'll only be working with a small number of repos, so we
can just list project-config, zuul, and nodepool.  We can probably put
this in project-config/zuul/main.yaml.

When listing each of the projects in main.yaml, we will specify whether
it is a "config repo" or a "project repo".  The distinction is explained
below in _Security Contexts_; for the moment, just know that this is
where we will configure that.

When Zuul reads this config file from the local disk, it then performs a
bunch of git operations to read the rest of its configuration from those
repos.  This is how the live dynamic reconfiguration is possible.

We will *also* have a significant chunk of Zuul configuration inside of
project-config that will be read this way.  It's worth noting that is
distinct from the tenant config.  These things don't have to be in the
same repo, I think that just makes the most sense for the way we are
organized in OpenStack.  The main part of the dynamic config will be in
project-config/zuul.yaml.  That is where we will define pipelines and
any centralized jobs.

Security Contexts
-

Whenever a playbook runs, it runs with a security context which is
determined by the repo in which the playbook resides.

Repositories which are designated as "project repos" are considered
untrusted.  This means that any playbook defined in that repo is not
permitted to access anything on the Zuul launcher.  It can use any
Ansible module on the remote worker nodes, but it can not, for example,
access local storage on the launcher itself, nor can it use non-standard
Ansible plugins.  Because of the relative safety that this affords,
project repos are permitted to change their configuration dynamically --
that is, a proposed change to a .zuul.yaml file within a project repo
will cause Zuul to run *with the proposed configuration change* when
evaluating it.

Repositories which are "config repos" are trusted and can contain jobs
and configuration which could potentially be used to compromise the Zuul
system.  Pre and Post Playbooks (more on this later) defined here *are*
permitted to access local storage on the Zuul launcher.  They can also
use ansible plugins which may allow even further local or network
actions.  Because of this extra level of access, proposed changes to
config repos are not run with the new configuration.  They retain their
current configuration until the proposed change is approved and actually
lands.  However, it's worth noting that since Zuul can detect that
event, it will be able to immediately reconfigure itself after a change
to a config repo lands.

Obviously, in OpenStack, people are pretty excited about being able to
see job configuration changes run before they land.  Any jobs which are
configured in project-config won't be able to take advantage of that
(though, because of the immediate reconfiguration, iteration will be
much faster than before).  To deal with that, we can restrict ourselves
to defining only the jobs or job components which require extra care or
levels of access in project-config.  The remainder of the configuration
can either be placed in the projects themselves, or perhaps in a newly
created central repo similar to project-config, but without the same
level of access.

As an example, the configuration of the basic devstack job

Re: [OpenStack-Infra] Ask.o.o down

[Jeremy Stanley]

On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote:
[...]
> Down again, this time with "Network is unreachable".
[...]

I'm not finding any obvious errors on the server nor relevant
maintenance notices/trouble tickets from the service provider to
explain this. I do see conspicuous gaps in network traffic volume
and system load from ~06:45 to ~08:10 UTC according to cacti:

http://cacti.openstack.org/?tree_id=1&leaf_id=156

Skipping back through previous days I find some similar gaps
starting anywhere from 06:30 to 07:00 and ending between 07:00 and
08:00 but they don't seem to occur every day and I'm not having much
luck finding a pattern. It _is_ conspicuously close to when
/etc/cron.daily scripts get fired from the crontab so might coincide
with log rotation/service restarts? The graphs don't show these gaps
correlating with any spikes in CPU, memory or disk activity so it
doesn't seem to be resource starvation (at least not for any common
resources we're tracking).
-- 
Jeremy Stanley

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Ask.o.o down

[Tom Fifield]

On 10/02/17 16:41, Marton Kiss wrote:

Hi Tom

Still not working from your end? Works perfectly from here.


It is back now.



Marton

On Fri, Feb 10, 2017 at 9:14 AM Tom Fifield mailto:t...@openstack.org>> wrote:

On 10/02/17 16:08, Tom Fifield wrote:
> On 13/01/17 14:41, Tom Fifield wrote:
>> As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on
>> IPv4) - tried from a couple of different hosts
>>
>> fifieldt@docwork2:~$ wget ask.openstack.org

>> --2017-01-13 06:40:49--  http://ask.openstack.org/
>> Resolving ask.openstack.org 
(ask.openstack.org )... 23.253.72.95,
>> 2001:4800:7815:103:be76:4eff:fe05:89f3
>> Connecting to ask.openstack.org 
(ask.openstack.org )|23.253.72.95|:80...
>> failed: Connection refused.
>
> Down again, this time with "Network is unreachable".

Sorry, it is still connection refused as before. The network unreachable
bit is my end's IPv6 connectivity :)


___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra




___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Ask.o.o down

[Marton Kiss]

Hi Tom

Still not working from your end? Works perfectly from here.

Marton

On Fri, Feb 10, 2017 at 9:14 AM Tom Fifield  wrote:

> On 10/02/17 16:08, Tom Fifield wrote:
> > On 13/01/17 14:41, Tom Fifield wrote:
> >> As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on
> >> IPv4) - tried from a couple of different hosts
> >>
> >> fifieldt@docwork2:~$ wget ask.openstack.org
> >> --2017-01-13 06:40:49--  http://ask.openstack.org/
> >> Resolving ask.openstack.org (ask.openstack.org)... 23.253.72.95,
> >> 2001:4800:7815:103:be76:4eff:fe05:89f3
> >> Connecting to ask.openstack.org (ask.openstack.org)|23.253.72.95|:80...
> >> failed: Connection refused.
> >
> > Down again, this time with "Network is unreachable".
>
> Sorry, it is still connection refused as before. The network unreachable
> bit is my end's IPv6 connectivity :)
>
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Ask.o.o down

[Tom Fifield]

On 13/01/17 14:41, Tom Fifield wrote:

As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on
IPv4) - tried from a couple of different hosts

fifieldt@docwork2:~$ wget ask.openstack.org
--2017-01-13 06:40:49--  http://ask.openstack.org/
Resolving ask.openstack.org (ask.openstack.org)... 23.253.72.95,
2001:4800:7815:103:be76:4eff:fe05:89f3
Connecting to ask.openstack.org (ask.openstack.org)|23.253.72.95|:80...
failed: Connection refused.


Down again, this time with "Network is unreachable".


fifieldt@docwork2:~$ wget ask.openstack.org
--2017-02-10 08:07:10--  http://ask.openstack.org/
Resolving ask.openstack.org (ask.openstack.org)... 23.253.72.95, 
2001:4800:7815:103:be76:4eff:fe05:89f3
Connecting to ask.openstack.org (ask.openstack.org)|23.253.72.95|:80... 
failed: Connection refused.
Connecting to ask.openstack.org 
(ask.openstack.org)|2001:4800:7815:103:be76:4eff:fe05:89f3|:80... 
failed: Network is unreachable.




fifieldt@docwork2:~$ traceroute ask.openstack.org
traceroute to ask.openstack.org (23.253.72.95), 30 hops max, 60 byte packets
[snip]
11  xe-0-2-4.bdr1.a.sjc.aarnet.net.au (202.158.194.162)  160.649 ms 
160.670 ms  160.560 ms
12  xe-0-8-0-21.r01.snjsca04.us.bb.gin.ntt.net (128.241.219.153) 
161.276 ms  161.056 ms  161.216 ms
13  ae-10.r23.snjsca04.us.bb.gin.ntt.net (129.250.3.174)  161.199 ms 
161.050 ms  161.157 ms
14  ae-7.r23.dllstx09.us.bb.gin.ntt.net (129.250.4.155)  200.216 ms 
200.287 ms  200.261 ms
15  ae-26.r01.dllstx04.us.bb.gin.ntt.net (129.250.6.129)  199.711 ms 
199.693 ms  200.515 ms
16  ae-0.rackspace.dllstx04.us.bb.gin.ntt.net (129.250.207.118)  199.738 
ms  199.764 ms  199.303 ms

17  * * *
18  74.205.108.121 (74.205.108.121)  200.814 ms 
be41.coreb.dfw1.rackspace.net (74.205.108.117)  201.289 ms  202.648 ms
19  po1.corea-core10.core10.dfw3.rackspace.net (74.205.108.49)  201.402 
ms  202.792 ms po2.coreb-core10.core10.dfw3.rackspace.net 
(74.205.108.51)  201.389 ms
20  core9.aggr160a-4.dfw2.rackspace.net (98.129.84.201)  205.857 ms 
core9.aggr160b-4.dfw2.rackspace.net (98.129.84.203)  203.025 ms  204.921 ms
21  ask.openstack.org (23.253.72.95)  200.853 ms !X  202.494 ms !X 
200.562 ms !X



fifieldt@usagi:~$ traceroute ask.openstack.org
traceroute to ask.openstack.org (23.253.72.95), 30 hops max, 60 byte packets
[snip]
12  las-b21-link.telia.net (62.115.125.142)  137.764 ms  136.495 ms 
133.458 ms
13  dls-b21-link.telia.net (62.115.123.138)  203.140 ms 
dls-b21-link.telia.net (62.115.123.136)  166.356 ms 
dls-b21-link.telia.net (62.115.122.20)  168.077 ms
14  rackspace-ic-302090-dls-bb1.c.telia.net (62.115.33.78)  189.567 ms 
187.802 ms  178.015 ms
15  10.25.1.71 (10.25.1.71)  188.676 ms  184.027 ms 10.25.1.66 
(10.25.1.66)  172.849 ms
16  74.205.108.121 (74.205.108.121)  167.863 ms 
be41.corea.dfw1.rackspace.net (74.205.108.113)  186.449 ms 
74.205.108.121 (74.205.108.121)  167.069 ms
17  po1.corea-core9.core9.dfw3.rackspace.net (74.205.108.45)  204.053 ms 
 204.352 ms  182.891 ms
18  core9.aggr160b-4.dfw2.rackspace.net (98.129.84.203)  211.036 ms 
core10.aggr160b-4.dfw2.rackspace.net (98.129.84.207)  206.673 ms 
core10.aggr160a-4.dfw2.rackspace.net (98.129.84.205)  178.904 ms
19  ask.openstack.org (23.253.72.95)  200.364 ms !X  201.845 ms !X 
186.189 ms !X



___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Ask.o.o down

[Tom Fifield]

On 10/02/17 16:08, Tom Fifield wrote:

On 13/01/17 14:41, Tom Fifield wrote:

As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on
IPv4) - tried from a couple of different hosts

fifieldt@docwork2:~$ wget ask.openstack.org
--2017-01-13 06:40:49--  http://ask.openstack.org/
Resolving ask.openstack.org (ask.openstack.org)... 23.253.72.95,
2001:4800:7815:103:be76:4eff:fe05:89f3
Connecting to ask.openstack.org (ask.openstack.org)|23.253.72.95|:80...
failed: Connection refused.


Down again, this time with "Network is unreachable".


Sorry, it is still connection refused as before. The network unreachable 
bit is my end's IPv6 connectivity :)



___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra