Re: [OpenStack-Infra] Zuul roadmap

[Melvin Hillsman]

Hey Paul,

I am also interested in helping here; will read over the etherpad today just to 
make sure I am caught up.

-- 
Kind regards,

Melvin Hillsman
mrhills...@gmail.com
mobile: +1 (832) 264-2646
irc: mrhillsman

On 12/12/17, 10:29 AM, "Paul Belanger"  wrote:

On Tue, Dec 12, 2017 at 05:01:41PM +0100, Matthieu Huin wrote:
> Hello,
> 
> If the getting-started documentation effort is also aimed at end
> users, I'd be happy to help Leif with this: we've written a quick
> start guide for Software Factory explaining how to set up pipelines
> and jobs with Zuul3, and this would probably be better hosted upstream
> with minimal adaptations. Let me know if there's interest for this
> (the storyboard item at
> https://storyboard.openstack.org/#!/story/2001332 doesn't specify
> which kind of doc is expected) and I can submit some patches to the
> documentatoin.
> 
I was just talking with leifmadsen about it this morning and we're going to
organize a working group on docs in the coming days. With holidays coming up
quick, it might be difficult to wrap things up before Christmas.

I know there already has been some discussion between Jim and Leif, plus 
myself
and Leif documented in the etherpad[1]. Using Fedora and github, I believe 
the
etherpad notes are correct. So the next steps are reformatting into RST and
tuning the docs.

TL;DR we have some docs, and jobs ran, now to make that a little more user
friendly.

[1] https://etherpad.openstack.org/p/zuulv3-quickstart

> Best regards,
> 
> MHU
> 
> On Fri, Dec 8, 2017 at 9:25 PM, David Shrewsbury
>  wrote:
> > Hi!
> >
> > On Wed, Dec 6, 2017 at 10:34 AM, James E. Blair  
wrote:
> >
> > 
> >
> >>
> >> * Add finger gateway
> >>
> >> The fact that the executor must be started as root in order to listen 
on
> >> port 79 is awkward for new users.  It can be avoided by configuring it
> >> to listen on a different port, but that's also awkward.  In either 
case,
> >> it's a significant hurdle, and it's one of the most frequently asked
> >> questions in IRC.  The plan to deal with this is to create a new 
service
> >> solely to multiplex the finger streams.  This is very similar to the
> >> zuul-web service for multiplexing the console streams, so all the
> >> infrastructure is in place.  And of course, running this service will 
be
> >> optional, so it means that new users don't even have to deal with it to
> >> get up and running, like they do now.  Adding a new service to the 3.0
> >> list should not be done lightly, but the improvement in experience for
> >> new users will be significant.
> >>
> >> David Shrewsbury has started on this.  I don't think it is out of 
reach.
> >
> >
> >
> > Indeed, it is not out of reach:
> >
> >https://review.openstack.org/525276
> >
> >
> >
> > --
> > David Shrewsbury (Shrews)
> >
> > ___
> > OpenStack-Infra mailing list
> > OpenStack-Infra@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> 
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra



___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Paul Belanger]

On Tue, Dec 12, 2017 at 05:01:41PM +0100, Matthieu Huin wrote:
> Hello,
> 
> If the getting-started documentation effort is also aimed at end
> users, I'd be happy to help Leif with this: we've written a quick
> start guide for Software Factory explaining how to set up pipelines
> and jobs with Zuul3, and this would probably be better hosted upstream
> with minimal adaptations. Let me know if there's interest for this
> (the storyboard item at
> https://storyboard.openstack.org/#!/story/2001332 doesn't specify
> which kind of doc is expected) and I can submit some patches to the
> documentatoin.
> 
I was just talking with leifmadsen about it this morning and we're going to
organize a working group on docs in the coming days. With holidays coming up
quick, it might be difficult to wrap things up before Christmas.

I know there already has been some discussion between Jim and Leif, plus myself
and Leif documented in the etherpad[1]. Using Fedora and github, I believe the
etherpad notes are correct. So the next steps are reformatting into RST and
tuning the docs.

TL;DR we have some docs, and jobs ran, now to make that a little more user
friendly.

[1] https://etherpad.openstack.org/p/zuulv3-quickstart

> Best regards,
> 
> MHU
> 
> On Fri, Dec 8, 2017 at 9:25 PM, David Shrewsbury
>  wrote:
> > Hi!
> >
> > On Wed, Dec 6, 2017 at 10:34 AM, James E. Blair  wrote:
> >
> > 
> >
> >>
> >> * Add finger gateway
> >>
> >> The fact that the executor must be started as root in order to listen on
> >> port 79 is awkward for new users.  It can be avoided by configuring it
> >> to listen on a different port, but that's also awkward.  In either case,
> >> it's a significant hurdle, and it's one of the most frequently asked
> >> questions in IRC.  The plan to deal with this is to create a new service
> >> solely to multiplex the finger streams.  This is very similar to the
> >> zuul-web service for multiplexing the console streams, so all the
> >> infrastructure is in place.  And of course, running this service will be
> >> optional, so it means that new users don't even have to deal with it to
> >> get up and running, like they do now.  Adding a new service to the 3.0
> >> list should not be done lightly, but the improvement in experience for
> >> new users will be significant.
> >>
> >> David Shrewsbury has started on this.  I don't think it is out of reach.
> >
> >
> >
> > Indeed, it is not out of reach:
> >
> >https://review.openstack.org/525276
> >
> >
> >
> > --
> > David Shrewsbury (Shrews)
> >
> > ___
> > OpenStack-Infra mailing list
> > OpenStack-Infra@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> 
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Matthieu Huin]

Hello,

If the getting-started documentation effort is also aimed at end
users, I'd be happy to help Leif with this: we've written a quick
start guide for Software Factory explaining how to set up pipelines
and jobs with Zuul3, and this would probably be better hosted upstream
with minimal adaptations. Let me know if there's interest for this
(the storyboard item at
https://storyboard.openstack.org/#!/story/2001332 doesn't specify
which kind of doc is expected) and I can submit some patches to the
documentatoin.

Best regards,

MHU

On Fri, Dec 8, 2017 at 9:25 PM, David Shrewsbury
 wrote:
> Hi!
>
> On Wed, Dec 6, 2017 at 10:34 AM, James E. Blair  wrote:
>
> 
>
>>
>> * Add finger gateway
>>
>> The fact that the executor must be started as root in order to listen on
>> port 79 is awkward for new users.  It can be avoided by configuring it
>> to listen on a different port, but that's also awkward.  In either case,
>> it's a significant hurdle, and it's one of the most frequently asked
>> questions in IRC.  The plan to deal with this is to create a new service
>> solely to multiplex the finger streams.  This is very similar to the
>> zuul-web service for multiplexing the console streams, so all the
>> infrastructure is in place.  And of course, running this service will be
>> optional, so it means that new users don't even have to deal with it to
>> get up and running, like they do now.  Adding a new service to the 3.0
>> list should not be done lightly, but the improvement in experience for
>> new users will be significant.
>>
>> David Shrewsbury has started on this.  I don't think it is out of reach.
>
>
>
> Indeed, it is not out of reach:
>
>https://review.openstack.org/525276
>
>
>
> --
> David Shrewsbury (Shrews)
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[David Shrewsbury]

Hi!

On Wed, Dec 6, 2017 at 10:34 AM, James E. Blair  wrote:




> * Add finger gateway
>
> The fact that the executor must be started as root in order to listen on
> port 79 is awkward for new users.  It can be avoided by configuring it
> to listen on a different port, but that's also awkward.  In either case,
> it's a significant hurdle, and it's one of the most frequently asked
> questions in IRC.  The plan to deal with this is to create a new service
> solely to multiplex the finger streams.  This is very similar to the
> zuul-web service for multiplexing the console streams, so all the
> infrastructure is in place.  And of course, running this service will be
> optional, so it means that new users don't even have to deal with it to
> get up and running, like they do now.  Adding a new service to the 3.0
> list should not be done lightly, but the improvement in experience for
> new users will be significant.
>
> David Shrewsbury has started on this.  I don't think it is out of reach.
>


Indeed, it is not out of reach:

   https://review.openstack.org/525276



-- 
David Shrewsbury (Shrews)
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Melvin Hillsman]

I am definitely working to learn more but if we can do any testing within 
OpenLab to help speed things along please let me know.

-- 
Kind regards,

Melvin Hillsman
mrhills...@gmail.com
mobile: +1 (832) 264-2646
irc: mrhillsman

On 12/6/17, 10:15 AM, "Jeremy Stanley"  wrote:

On 2017-12-06 07:34:06 -0800 (-0800), James E. Blair wrote:
[...]
> Having gone through that, there are some things we need to add to the
> list:
[...]

One more came up just a few minutes ago:

* Add a minimal base job to the standard library

I've volunteered to knock that out; should be fairly trivial but
also very important to have out of the way before we tell people
this is really turn-key.
-- 
Jeremy Stanley
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra



___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Jeremy Stanley]

On 2017-12-06 07:34:06 -0800 (-0800), James E. Blair wrote:
[...]
> Having gone through that, there are some things we need to add to the
> list:
[...]

One more came up just a few minutes ago:

* Add a minimal base job to the standard library

I've volunteered to knock that out; should be fairly trivial but
also very important to have out of the way before we tell people
this is really turn-key.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[James E. Blair]

Clint Byrum  writes:

> I know a bunch of this stuff is janky as all get out, because as much of the
> jankiness is my own fault as anybody else's. But so much work has gone into
> zuulv3 beyond what OpenStack needs, I am still not convinced we need to wait
> for any of this. Maybe the zuul-web stuff, since changing URLs after a release
> is going to be a bear.
>
> I'm confident we'll get some of these done soon, and I may even get a chance 
> to
> contribute directly. But we all know that complexity creeps into engineering 
> in
> the most frustrating ways. I'd prefer that this list gets pared down, and that
> the release comes basically at or right before PTG, even if this list doesn't
> all happen.

I understand where you're coming from, but please also understand that
folks are already starting to show up in #zuul asking us the same
questions repeatedly because of how janky it is.  We're getting really
close to a point where we're spending too much time talking about how
things are going to get a lot easier for folks in just a few weeks
instead of actually doing those things.  So let me go through the list
and either expand on why I think a thing is important for the release,
or move it down in priority.

>> * granular quota support in nodepool (tobias)
>> * zuul-web dashboard (tristanC)
>> * update private key api for zuul-web (jeblair)

These things are basically done.  I agree they don't have to block the
release, but they are so likely to land very soon, we should just plan
for that.  If they don't we won't wait for them.

>> * github event ingestion via zuul-web (jlk)

Zuul currently has two web servers, and telling people how to set both
of them up is complicated.  This is the sort of thing that will cause
people to think that either this software is not ready to use, or it's
too complicated.

>> * abstract flag (do not run this job) (jeblair)

(I have a WIP patch for this)

We can move this to v3.1.

>> * zuul_json fixes (dmsimard)

This is a known bug that causes Zuul to fail with certain perfectly
valid uses of Ansible.  It's easy for users to hit, but it should also
be easy to fix.

>> * refactor config loading (jeblair)

Originally this task was mostly about solving the forward inheritance
problem, which is done.  At this point, I consider the task to be more
akin to double checking that we aren't missing anything major from the
job language that we can't fix in the future.

>> * protected flag (inherit only within this project) (jeblair)

(Tobias has a patch for this)

We can move this to v3.1.

>> * refactor zuul_stream and add testing (mordred)

This is important because there are still a number of cases where errors
in Ansible are not reported in the streaming log.  We need to handle
those cases, but this code has evolved quite a bit from its original
implementation to the point where it is difficult to understand, and it
has very limited testing.  This module is nearly frozen until this
refactor happens.  This means that new users (the most likely to hit the
bugs currently masked by this) are going to have a frustrating time --
they'll have to go look at executor logs to identify job failures.

Having said that, if the release came down to this alone, we could
probably delay it.  I'd like to keep this on the list and prioritize
work on it so we can get it into v3.0, but I'm okay deferring it if it's
the last thing standing.

>> * getting-started documentation (leifmadsen)

This is also really important to have for folks -- when we release 3.0
and say "okay, we've spent 2 years telling you not to use it, go use it
now" we should have some instructions to help people do that.  It's a
complicated system, and I don't want folks bouncing off of it the first
time they try it.

However, I'll repeat the caveat from the last item here -- if it's the
last thing standing, we don't have to wait for it.

>> * demonstrate openstack-infra reporting on github
(pabelanger has since volunteered for this and begun work)

This item is about ensuring that the GitHub support works at scale.
We've had a number of folks using the GitHub driver, but as soon as we
started having the openstack-infra instance of Zuul watch some busy
GitHub repos, we started seeing errors in the log.  This is an important
new feature, and I want to make sure it's ready.  This item will either
be easy because we've already fixed the major issues, or we're going to
discover serious bugs that we would deal with soon after release anyway.

>> * cross-source dependencies

This is part of the work of adding a second source.  One of Zuul's major
features is cross-repo dependencies, and when we release GitHub support
for Zuul, I think it's important that we're able to tell a story about
how Zuul can integrate all of the repositories it works with.  It is so
much more compelling.  I understand that some folks don't need this, but
for a lot of the folks interested in Zuul and awaiting the 3.0 release
it is.

>> * add 

Re: [OpenStack-Infra] Zuul roadmap

[James E. Blair]

Fabien Boucher  writes:

>> * finish git driver
>>
>
> If ok for you, I want to propose myself to work on that git driver topic.
> I'll try to
> provide a first patch asap.

That's great, thanks!

There's some code there already, but it has no tests and hasn't been
used in a while -- I don't know if it still works.

Some brief thoughts about it:

* We should add some tests that exercise it
* It currently only supports file:// urls; it should also support
  https://
* It should periodically poll for new updates (with a configurable
  interval, maybe default to 2 hours?)
* When a branch is updated, it should perform a diffstat to determine if
  any zuul config files were updated, and if so, emit an event for the
  scheduler to reconfigure that tenant

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Fabien Boucher]

Hi,

On Wed, Nov 1, 2017 at 10:47 PM, James E. Blair  wrote:

> Hi,
>
> At the PTG we brainstormed a road map for Zuul once we completed the
> infra cutover.  I think we're in a position now that we can get back to
> thinking about this, so I've (slightly) cleaned it up and organized it
> here.
>
> I've grouped into a number of sections.  First:
>
> Very Near Term
> --
>
> These are things that we should be able to land within just a few weeks
> at most, once we're back from the OpenStack summit and can pay more
> attention to work other than the openstack-infra migration.  All of
> these are already in progress (some are basically finished) and all have
> a primary driver assigned:
>
> * granular quota support in nodepool (tobias)
> * zuul-web dashboard (tristanC)
> * update private key api for zuul-web (jeblair)
> * github event ingestion via zuul-web (jlk)
> * abstract flag (do not run this job) (jeblair)
> * zuul_json fixes (dmsimard)
>
> Short Term
> --
>
> These are things we should be able to do within the weeks or months
> following.  Some have had work start on them already and have a driver
> assigned, others are still up for grabs.  These are things we really
> ought to get done before the v3.0 release because either they involve
> some of the defining features of v3, make it possible to actually deploy
> and run v3, or may involve significant changes for which we don't want
> to have to deal with backwards compatability.
>
> * refactor config loading (jeblair)
> * protected flag (inherit only within this project) (jeblair)
> * refactor zuul_stream and add testing (mordred)
> * getting-started documentation (leifmadsen)
> * demonstrate openstack-infra reporting on github
> * cross-source dependencies
> * add command socket to scheduler and merger for consistent start/stop
> * finish git driver
>

If ok for you, I want to propose myself to work on that git driver topic.
I'll try to
provide a first patch asap.


> * standardize javascript tooling
>
> -- v3.0 release 
>
> Yay!  After we release...
>
> Medium Term
> ---
>
> Once the initial v3 release is out the door, there are some things that
> we have been planning on for a while and should work on to improve the
> v3 story.  These should be straightforward to implement, but these don't
> need to hold up the release and can easily fit into v3.1.
>
> * add line comment support to reporters
> * gerrit ci reporting (2.14)
> * add cleanup jobs (jobs that always run even if parents fail)
> * automatic job doc generation
>
> Long Term / Design
> --
>
> Some of these are items that we should either discuss a bit further
> before implementing, but most of them probably warrant an proposal in
> infra-specs so we can flesh out the design before we start work.
>
> * gerrit ingestion via separate process?
> * per-job artifact location
> * need way for admin to trigger a single job (not just a buildset)
> * nodepool backends
> * nodepool label access (tenant/project label restrictions?)
> * nodepool tenant awareness?
> * nodepool rest api alignment?
> * selinux domains
> * fedmesg driver (trigger/reporter)
> * mqtt driver (trigger/reporter)
> * nodepool status ui?
>
> How does this look?
>
> -Jim
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Monty Taylor]

On 11/21/2017 09:08 AM, Paul Belanger wrote:

On Wed, Nov 01, 2017 at 02:47:20PM -0700, James E. Blair wrote:

Hi,

At the PTG we brainstormed a road map for Zuul once we completed the
infra cutover.  I think we're in a position now that we can get back to
thinking about this, so I've (slightly) cleaned it up and organized it
here.

I've grouped into a number of sections.  First:

Very Near Term
--

These are things that we should be able to land within just a few weeks
at most, once we're back from the OpenStack summit and can pay more
attention to work other than the openstack-infra migration.  All of
these are already in progress (some are basically finished) and all have
a primary driver assigned:

* granular quota support in nodepool (tobias)
* zuul-web dashboard (tristanC)
* update private key api for zuul-web (jeblair)
* github event ingestion via zuul-web (jlk)
* abstract flag (do not run this job) (jeblair)
* zuul_json fixes (dmsimard)

Short Term
--

These are things we should be able to do within the weeks or months
following.  Some have had work start on them already and have a driver
assigned, others are still up for grabs.  These are things we really
ought to get done before the v3.0 release because either they involve
some of the defining features of v3, make it possible to actually deploy
and run v3, or may involve significant changes for which we don't want
to have to deal with backwards compatability.

* refactor config loading (jeblair)
* protected flag (inherit only within this project) (jeblair)
* refactor zuul_stream and add testing (mordred)
* getting-started documentation (leifmadsen)
* demonstrate openstack-infra reporting on github

I can start working on this one, is there any objections if we use
gtest-org/ansible first?


No objections from me. I've also got an integration test ready and 
waiting to go for shade + ansible/ansible whenever we're ready to do that.



* cross-source dependencies
* add command socket to scheduler and merger for consistent start/stop

I can see about working on this too


* finish git driver
* standardize javascript tooling

-- v3.0 release 

Yay!  After we release...

Medium Term
---

Once the initial v3 release is out the door, there are some things that
we have been planning on for a while and should work on to improve the
v3 story.  These should be straightforward to implement, but these don't
need to hold up the release and can easily fit into v3.1.

* add line comment support to reporters
* gerrit ci reporting (2.14)
* add cleanup jobs (jobs that always run even if parents fail)
* automatic job doc generation

Long Term / Design
--

Some of these are items that we should either discuss a bit further
before implementing, but most of them probably warrant an proposal in
infra-specs so we can flesh out the design before we start work.

* gerrit ingestion via separate process?
* per-job artifact location
* need way for admin to trigger a single job (not just a buildset)
* nodepool backends
* nodepool label access (tenant/project label restrictions?)
* nodepool tenant awareness?
* nodepool rest api alignment?
* selinux domains
* fedmesg driver (trigger/reporter)
* mqtt driver (trigger/reporter)
* nodepool status ui?

How does this look?

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra


___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra




___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Paul Belanger]

On Wed, Nov 01, 2017 at 02:47:20PM -0700, James E. Blair wrote:
> Hi,
> 
> At the PTG we brainstormed a road map for Zuul once we completed the
> infra cutover.  I think we're in a position now that we can get back to
> thinking about this, so I've (slightly) cleaned it up and organized it
> here.
> 
> I've grouped into a number of sections.  First:
> 
> Very Near Term
> --
> 
> These are things that we should be able to land within just a few weeks
> at most, once we're back from the OpenStack summit and can pay more
> attention to work other than the openstack-infra migration.  All of
> these are already in progress (some are basically finished) and all have
> a primary driver assigned:
> 
> * granular quota support in nodepool (tobias)
> * zuul-web dashboard (tristanC)
> * update private key api for zuul-web (jeblair)
> * github event ingestion via zuul-web (jlk)
> * abstract flag (do not run this job) (jeblair)
> * zuul_json fixes (dmsimard)
> 
> Short Term
> --
> 
> These are things we should be able to do within the weeks or months
> following.  Some have had work start on them already and have a driver
> assigned, others are still up for grabs.  These are things we really
> ought to get done before the v3.0 release because either they involve
> some of the defining features of v3, make it possible to actually deploy
> and run v3, or may involve significant changes for which we don't want
> to have to deal with backwards compatability.
> 
> * refactor config loading (jeblair)
> * protected flag (inherit only within this project) (jeblair)
> * refactor zuul_stream and add testing (mordred)
> * getting-started documentation (leifmadsen)
> * demonstrate openstack-infra reporting on github
I can start working on this one, is there any objections if we use
gtest-org/ansible first?

> * cross-source dependencies
> * add command socket to scheduler and merger for consistent start/stop
I can see about working on this too

> * finish git driver
> * standardize javascript tooling
> 
> -- v3.0 release 
> 
> Yay!  After we release...
> 
> Medium Term
> ---
> 
> Once the initial v3 release is out the door, there are some things that
> we have been planning on for a while and should work on to improve the
> v3 story.  These should be straightforward to implement, but these don't
> need to hold up the release and can easily fit into v3.1.
> 
> * add line comment support to reporters
> * gerrit ci reporting (2.14)
> * add cleanup jobs (jobs that always run even if parents fail)
> * automatic job doc generation
> 
> Long Term / Design
> --
> 
> Some of these are items that we should either discuss a bit further
> before implementing, but most of them probably warrant an proposal in
> infra-specs so we can flesh out the design before we start work.
> 
> * gerrit ingestion via separate process?
> * per-job artifact location
> * need way for admin to trigger a single job (not just a buildset)
> * nodepool backends
> * nodepool label access (tenant/project label restrictions?)
> * nodepool tenant awareness?
> * nodepool rest api alignment?
> * selinux domains
> * fedmesg driver (trigger/reporter)
> * mqtt driver (trigger/reporter)
> * nodepool status ui?
> 
> How does this look?
> 
> -Jim
> 
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Tony Breeds]

On Thu, Nov 02, 2017 at 12:33:30AM +, Tristan Cacqueray wrote:
> On November 1, 2017 11:56 pm, Tony Breeds wrote:
> > On Wed, Nov 01, 2017 at 02:47:20PM -0700, James E. Blair wrote:
> > > Hi,
> > > 
> > > At the PTG we brainstormed a road map for Zuul once we completed the
> > > infra cutover.  I think we're in a position now that we can get back to
> > > thinking about this, so I've (slightly) cleaned it up and organized it
> > > here.
> > 
> > 
> > > How does this look?
> > 
> > Something we talked about on IRC that I didnt' see there was a REST API
> > to get the current config for a project.  Something that requirements
> > and release management can use to validate that $project has the correct
> > templates in place for all the automation to work.
> > 
> > Right now we're just checking-project as assuming that's good enough and

checking-project ?  That should be 'checking project-config' :/

I can't even blame a lack of coffee.

> > it probably is ;P but It'd still be cool if the REST API was there at
> > some point.
> > 
> > Yours Tony.
> 
> That's a great idea, actually the zuul-web dashboard[0] mentioned in the
> "very near term" is actually a REST API from the zuul-web service.
> It currently only exposes jobs list and build result history, though it
> should be possible to expose a projects.json endpoint to give you that
> information. I think this would be a great addition to provide more visual
> feedback about project's pipeline and job dependencies.
> 
> Perhaps we would also need a templates.json endpoint too so that a
> project configuration would reference templates instead/alongside the
> expanded configuration.

That sounds good.  I'm sorry I didn't equate the dashboard and the REST
API.

Yours Tony.


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Tristan Cacqueray]

On November 1, 2017 11:56 pm, Tony Breeds wrote:

On Wed, Nov 01, 2017 at 02:47:20PM -0700, James E. Blair wrote:

Hi,

At the PTG we brainstormed a road map for Zuul once we completed the
infra cutover.  I think we're in a position now that we can get back to
thinking about this, so I've (slightly) cleaned it up and organized it
here.



 

How does this look?


Something we talked about on IRC that I didnt' see there was a REST API
to get the current config for a project.  Something that requirements
and release management can use to validate that $project has the correct
templates in place for all the automation to work.

Right now we're just checking-project as assuming that's good enough and
it probably is ;P but It'd still be cool if the REST API was there at
some point.

Yours Tony.


That's a great idea, actually the zuul-web dashboard[0] mentioned in the
"very near term" is actually a REST API from the zuul-web service.
It currently only exposes jobs list and build result history, though it
should be possible to expose a projects.json endpoint to give you that
information. I think this would be a great addition to provide more visual
feedback about project's pipeline and job dependencies.

Perhaps we would also need a templates.json endpoint too so that a
project configuration would reference templates instead/alongside the
expanded configuration.

Regards,
-Tristan

[0]: https://review.openstack.org/#/q/topic:zuul-web


pgpS7EV1IAEkK.pgp
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul roadmap

[Tony Breeds]

On Wed, Nov 01, 2017 at 02:47:20PM -0700, James E. Blair wrote:
> Hi,
> 
> At the PTG we brainstormed a road map for Zuul once we completed the
> infra cutover.  I think we're in a position now that we can get back to
> thinking about this, so I've (slightly) cleaned it up and organized it
> here.


 
> How does this look?

Something we talked about on IRC that I didnt' see there was a REST API
to get the current config for a project.  Something that requirements
and release management can use to validate that $project has the correct
templates in place for all the automation to work.

Right now we're just checking-project as assuming that's good enough and
it probably is ;P but It'd still be cool if the REST API was there at
some point.

Yours Tony.


signature.asc
Description: PGP signature
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] Zuul roadmap

[James E. Blair]

Hi,

At the PTG we brainstormed a road map for Zuul once we completed the
infra cutover.  I think we're in a position now that we can get back to
thinking about this, so I've (slightly) cleaned it up and organized it
here.

I've grouped into a number of sections.  First:

Very Near Term
--

These are things that we should be able to land within just a few weeks
at most, once we're back from the OpenStack summit and can pay more
attention to work other than the openstack-infra migration.  All of
these are already in progress (some are basically finished) and all have
a primary driver assigned:

* granular quota support in nodepool (tobias)
* zuul-web dashboard (tristanC)
* update private key api for zuul-web (jeblair)
* github event ingestion via zuul-web (jlk)
* abstract flag (do not run this job) (jeblair)
* zuul_json fixes (dmsimard)

Short Term
--

These are things we should be able to do within the weeks or months
following.  Some have had work start on them already and have a driver
assigned, others are still up for grabs.  These are things we really
ought to get done before the v3.0 release because either they involve
some of the defining features of v3, make it possible to actually deploy
and run v3, or may involve significant changes for which we don't want
to have to deal with backwards compatability.

* refactor config loading (jeblair)
* protected flag (inherit only within this project) (jeblair)
* refactor zuul_stream and add testing (mordred)
* getting-started documentation (leifmadsen)
* demonstrate openstack-infra reporting on github
* cross-source dependencies
* add command socket to scheduler and merger for consistent start/stop
* finish git driver
* standardize javascript tooling

-- v3.0 release 

Yay!  After we release...

Medium Term
---

Once the initial v3 release is out the door, there are some things that
we have been planning on for a while and should work on to improve the
v3 story.  These should be straightforward to implement, but these don't
need to hold up the release and can easily fit into v3.1.

* add line comment support to reporters
* gerrit ci reporting (2.14)
* add cleanup jobs (jobs that always run even if parents fail)
* automatic job doc generation

Long Term / Design
--

Some of these are items that we should either discuss a bit further
before implementing, but most of them probably warrant an proposal in
infra-specs so we can flesh out the design before we start work.

* gerrit ingestion via separate process?
* per-job artifact location
* need way for admin to trigger a single job (not just a buildset)
* nodepool backends
* nodepool label access (tenant/project label restrictions?)
* nodepool tenant awareness?
* nodepool rest api alignment?
* selinux domains
* fedmesg driver (trigger/reporter)
* mqtt driver (trigger/reporter)
* nodepool status ui?

How does this look?

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra