Re: [OpenStack-Infra] Zuul v3: proposed new Depends-On syntax

[Morgan Fainberg]

From someone who has used/consumed/worked with/contributed to Zuul,
this seems very straight forward and reasonable. The only concern is
ensuring that the metadata on "if this has in-fact merged" is clearly
available in the URL. I want to ensure we're implementing something
generally useful and if the url cannot provide data on "if this thing
is merged". As long as we have a clear "what we expect" and how to
implement new URL data providers (I'm sure gerrit and github
fundamentally provide different data in the web forms and we need to
suss out grabbing relevant data), this is perfect.

The downsides are relatively minimal (and could be solved with a
wrapper to look at git-notes).

I 100% agree we should not support all forms indefinitely, marking the
gerrit change-id form as deprecated is a solid plan when coupled with
this change.


On Wed, May 24, 2017 at 4:04 PM, James E. Blair  wrote:
> Hi,
>
> As part of Zuul v3, we're adding support for GitHub (and later possibly
> other systems).  We want these systems to have access to the full power
> of cross-project-dependencies in the same way as Gerrit.  However, the
> current syntax for the Depends-On footer is currently the
> Gerrit-specific change-id.
>
> We chose this in an attempt to be future-compatible with some proposed
> changes to Gerrit itself to support cross-project dependencies.  Since
> then, Gerrit has gone in a different direction on this subject, so I no
> longer think we should weigh that very heavily.
>
> While Gerrit change ids can be used to identify one or more changes
> within a Gerrit installation, there is no comparable identifier on
> GitHub, as pull request numbers are unique only within a project.
>
> The natural way to identify a GitHub pull request is with its URL.
>
> This can be used to identify Gerrit changes as well, and will likely be
> well supported by other systems.  Therefore, I propose we support URLs
> as the content of the Depends-On footers for all systems.  E.g.:
>
>   Depends-On: https://review.openstack.org/12345
>   Depends-On: https://github.com/ansible/ansible/pull/12345
>
> Similarly to the Gerrit change IDs, these identifiers are easily
> navigable within Gerrit (and Gertty), so that reviewers can traverse the
> dependency chain easily.
>
> One substantial aspect of this change is that it is more specific about
> projects and branches.  A single Gerrit change ID can refer to more than
> one branch, and even more than one project.  Zuul interprets this as
> "this change depends on *all* of the changes that match".  Often times
> that is convenient, but sometimes it is not.  Frequently users ask "how
> can I make this depend only on a change to master, not the backport of
> the change to stable?" and the answer is, "you can't".
>
> URLs have the advantage of allowing users to be specific as to which
> instances of a given change are actually required.  If, indeed, a change
> depends on more than one, of course a user can still add multiple
> Depends-On headers, one for each.
>
> It is also easy for Zuul connections to determine whether a given URL is
> referring to a change on that system without actually needing to query
> it.  A Zuul connected to several code review systems can easy determine
> which to ask for the change by examining the hostname.
>
> URLs do have two disadvantages compared to Gerrit change IDs: they can
> not be generated ahead of time, and they are not as easily found in
> offline git history.
>
> With Gerrit change IDs, we can write several local changes, and before
> pushing them to Gerrit, add Depends-On headers since the change id is
> generated locally.  URLs are not known until the changes are pushed to
> Gerrit (or GitHub pull requests opened).  So in some cases, editing of
> an already existing commit message may be required.  However, the most
> common case of a simple dependency chain can still be easily created by
> pushing one change up at a time.
>
> Change IDs, by virtue of being in the commit message of the dependent as
> well as depending change, become part of the permanent history of the
> project, no longer tied to the code review system, once they merge.
> This is an important thing to consider for long-running projects.  URLs
> are less suitable for this, since they acquire their context from
> contemporaneous servers.  However, Gerrit does record the review URL in
> git notes, so while it's not as convenient, with some additional tooling
> it should be possible to follow dependency paths with only the git
> history.
>
> Of course, this is not a change we can make instantaneously -- the
> change IDs have a lot of inertia and developer muscle memory.  And we
> don't want changes that have been in progress for a while to suddenly be
> broken with the switch to v3.  So we will need to support both syntaxes
> for some time.
>
> We could, indeed, support both syntaxes indefinitely, but I believe it
> would be better to plan on deprecating the Gerrit 

Re: [OpenStack-Infra] MeetBot taking unauthorised vacation

[Morgan Fainberg]

On Wed, Aug 17, 2016 at 8:57 AM, Anita Kuno  wrote:

> On 16-08-17 10:45 AM, Stig Telfer wrote:
>
>> I think our meeting ran over time and the openstack bot exited at 1
>> minute past the end of the meeting.  So it’s quite likely there were
>> supposed to be no meetings happening at that time, although in reality we
>> were still at it.
>>
>> Best wishes,
>> Stig
>>
>>
>> On 17 Aug 2016, at 15:09, David Moreau Simard  wrote:
>>>
>>> I love this.
>>>
>>> But on a serious note, the bot has to be restarted when commits are
>>> done to it and usually cores are quite dilligent about not merging
>>> reviews while there are meetings in progress.
>>> Maybe something slipped this time ?
>>>
>>> David Moreau Simard
>>> Senior Software Engineer | Openstack RDO
>>>
>>> dmsimard = [irc, github, twitter]
>>>
>>>
>>> On Wed, Aug 17, 2016 at 6:14 AM, Stig Telfer 
>>> wrote:
>>>
 Hi All -

 We just had a Scientific WG meeting in #openstack-meeting but the
 MeetBot left half way through.  Now the meeting cannot end (I’m sure it
 often feels that way).

 Best wishes,
 Stig


 ___
 OpenStack-Infra mailing list
 OpenStack-Infra@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

>>>
>> ___
>> OpenStack-Infra mailing list
>> OpenStack-Infra@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>
> The last change to the file that contains the list of channels for meetbot
> merged yesterday at 15:23 GMT: http://git.openstack.org/cgit/
> openstack-infra/system-config/commit/hiera/common.yaml?id=6d
> 3fc364f1f0d012672209b844a76b9554e916ea
>
> So I don't think this theory has legs.
>
> Bots disappear for all sorts of reasons, including code changes, I'm glad
> this one came back so you could end the meeting.
>
> Thank you,
> Anita.
>
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>

Often netsplits (even minor) will make the bots go haywire. Sometimes
netsplits recover on their own.
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] [gear] Making Gear easier to consume ( less .encode() and .decode() )

[Morgan Fainberg]

On Tue, Jun 21, 2016 at 3:16 PM, James E. Blair <cor...@inaugust.com> wrote:

> Morgan Fainberg <morgan.fainb...@gmail.com> writes:
>
> > As I have been converting Zuul and NodePool to python3, I have had to do
> a
> > bunch of changes around encode() and decode() of strings since gear is
> > (properly) an implementation of a protocol that requires binary data
> > (rather than text_strings).
> >
> > What this has highlighted is that gear should be made a bit more friendly
> > to use in the python world. We already explicitly assume a utf-8 encoding
> > for when things are turned into "binary" when crafting the job object in
> > certain cases [1].  I have discussed this with Jim Blair, and we both
> agree
> > that the ability to still reference attributes such as "job.name" (in a
> > simple manner that is straight forward), is important.
>
> Thanks for this email -- with this and a little browsing of the gear
> code, I think I've absorbed most of the context and am able to process
> this.  :)
>
> > Here is the outline of the change I'm proposing:
> >
> > * The main consumable part of gear (public classes) will convert the
> > "string" data we assign ( name[2], unique[3]) into utf-8-encoded bytes
> via
> > @property, @property.setter, and @property.getter for public consumption.
>
> This seems good.  I think we can make the assertion that in the API
> these things should always be utf-8 encoded strings, and do what's
> needed behind the scenes to encode/decode to bytes.
>
> I think this is the bulk of what we need to do to make gear
> user-friendly.
>
>
++ This is where I am going to focus.


> > * Arguments are explicitly supposed to be a binary_blob [4]. I am unsure
> if
> > this should also be automatically converted *or* if it should be the
> > inverse, .arguments and .arguments_string ?
>
> This is the thing we can't handle automatically.  At least, we can on
> the input side (if the user passes in a string, we could auto-encode but
> leave bytes alone -- but on the output side since the protocol is
> un-typed, we wouldn't know what to do).  So maybe this is a place where
> we should force the user to encode/decode.
>
> Looking at the gear code, I think that was the intent in a lot of
> places, but I think we may have gotten ahead of ourselves and written
> code assuming that isinstance('string', bytes) is False (which is the
> case in python3, but not in python2):
>
>
> https://github.com/openstack-infra/gear/blob/59d29104cb9c370e49b44f313e568bd43b172e1b/gear/__init__.py#L589
>
> That line of code explains a lot to me.
>
> I think we should stick close to the idea that arguments are binary
> blobs.
>
> From that, it seems to follow that reading job.arguments should get you
> a bytes object.
>
> On the one hand, I don't think it's unreasonable to say that users must
> encode/decode on that boundary.
>
> On the other hand, I think we could easily have a job.arguments_string
> property that decodes that from utf-8 for convenience, so why not?  I
> think I'm in favor of this.
>
> If we do that, then I think the question is: do we have gear
> automatically encode job arguments (in the Job constructor) if the user
> passes in a string?  I tend to favor this as well because it should
> almost always do the right thing in both python2 and python3 without
> inhibiting usage for either strings or binary data.  Does that sound
> reasonable?
>
>
I've spent a lot of time thinking about this, I'm going to say that I
agree, but I
feel like the _string property will be mostly unused. I want to see how
things look
before we add it in. If it makes things a lot better we can always add it.


> > * Internally gear will reference the encoded bits via the underlying
> > _binary form, which will allow direct access to a non-"string" form
> > of the data itself in the case that there is interesting things that need
> > to be handled via binary packing (for example) instead of "stringified"
> > versions.
>
> Sounds good.  Maybe we should call it "_bytes" though for clarity?
> /bikeshed
>
>
HAH! Sure. But as long as the bikeshed is cherenkov radiation blue.


> > * For compatibility the main @property.setter will handle either
> > binary_type or string_type (so we don't break everyone).
>
> Most of these are set in the constructor -- having a setter do that
> should make the constructor do the right thing, but in case there are
> cases where we need to remove type validation, I think it's worth saying
> that our intent is to have the constructor handle both forms as well for
> the same reason.
&

[OpenStack-Infra] [gear] Making Gear easier to consume ( less .encode() and .decode() )

[Morgan Fainberg]

As I have been converting Zuul and NodePool to python3, I have had to do a
bunch of changes around encode() and decode() of strings since gear is
(properly) an implementation of a protocol that requires binary data
(rather than text_strings).

What this has highlighted is that gear should be made a bit more friendly
to use in the python world. We already explicitly assume a utf-8 encoding
for when things are turned into "binary" when crafting the job object in
certain cases [1].  I have discussed this with Jim Blair, and we both agree
that the ability to still reference attributes such as "job.name" (in a
simple manner that is straight forward), is important.

Here is the outline of the change I'm proposing:

* The main consumable part of gear (public classes) will convert the
"string" data we assign ( name[2], unique[3]) into utf-8-encoded bytes via
@property, @property.setter, and @property.getter for public consumption.

* Arguments are explicitly supposed to be a binary_blob [4]. I am unsure if
this should also be automatically converted *or* if it should be the
inverse, .arguments and .arguments_string ?

* Internally gear will reference the encoded bits via the underlying
_binary form, which will allow direct access to a non-"string" form
of the data itself in the case that there is interesting things that need
to be handled via binary packing (for example) instead of "stringified"
versions.

* For compatibility the main @property.setter will handle either
binary_type or string_type (so we don't break everyone).

* The "_binary" will enforce that the data with be a binary_type
only.


I think this can be done in a single release of gear with minimal impact to
those using it. For what it is worth, it is unlikely that anyone has used
gear extensively in python3 as of yet because of recent bug fixes that
addressed py2->py3 compat issues around dict.values() and similar list() ->
iter() changes.

See the one question in the above proposal for "arguments".

[1]
https://github.com/openstack-infra/gear/blob/59d29104cb9c370e49b44f313e568bd43b172e1b/gear/__init__.py#L86
[2]
https://github.com/openstack-infra/gear/blob/59d29104cb9c370e49b44f313e568bd43b172e1b/gear/__init__.py#L2054
[3]
https://github.com/openstack-infra/gear/blob/59d29104cb9c370e49b44f313e568bd43b172e1b/gear/__init__.py#L2058
[4]
https://github.com/openstack-infra/gear/blob/59d29104cb9c370e49b44f313e568bd43b172e1b/gear/__init__.py#L2056

Thanks,
--Morgan
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] Please add me to the new ldappool gerrit groups.

[Morgan Fainberg]

Hi Openstack-Infra,

Please at your earliest convenience add me to the new ldapppol groups in
gerrit:

My username: mdrnstm
Groups:

* ldappool-core https://review.openstack.org/#/admin/groups/1389
* ldappool-release https://review.openstack.org/#/admin/groups/1390

I'll work with the appropriate teams to get those filled out once this is
done.


Thanks!
--Morgan
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] CentOS 7 AFS mirror now live

[Morgan Fainberg]

On Wed, May 11, 2016 at 12:51 PM, Paul Belanger 
wrote:

> I am happy to report our CentOS 7 AFS mirrors are now live[1]! Today we
> are only
> mirroring extras, os and updates. I don't think we need anything else, but
> if we
> do please let us know.
>
> The neat part of our CentOS mirrors is we still have gpgcheck enabled and
> using
> the RPM-GPG-KEY-CentOS-7 gpgkey. Meaning we are using properly signed
> RPMs. The
> reason we get this is because we are using rsync.
>
> Starting tomorrow, our centos-7 DIBs will enable our new AFS mirrors. I
> don't
> expect an issue, but if there are problems, we'll be able to revert using
> nodepool.
>
> Lastly, the only step left to complete is writing a validation script to
> parse
> the repodata files and ensure we've rsync'd all the repo files correctly
> before
> we `vos release mirror.centos` to AFS.
>
> [1] http://mirror.dfw.rax.openstack.org/centos/7/
> [2] https://review.openstack.org/#/c/315162/
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>

This is fantastic news! Great work!

--Morgan
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] [OT] Call for speaker submissions

[Morgan Fainberg]

On Tue, May 3, 2016 at 10:19 AM, Paul Belanger 
wrote:

> On Tue, May 03, 2016 at 10:01:06AM -0700, Elizabeth K. Joseph wrote:
> > On Mon, May 2, 2016 at 11:55 AM, Spencer Krum 
> wrote:
> > > I think this is a good conversation to have. I would recommend we start
> > > a thread-per-conference for simplicity. Folks can use this to
> coordinate
> > > attendance, ensure we submit unique presentations, and to help each
> > > other groom presentations. I doubt that anyone on the team would feel
> > > any offense at someone else submitting on infra topics, but
> coordination
> > > sounds good. Whenever we are presenting on an infra topic, being clear
> > > to put credit for building and supporting the technology in the correct
> > > place is something we should be careful to do.
> > >
> > > We've always used the publications repo to archive an collaborate on
> > > talks, but I think our use of that isn't quite correct at this time.
> >
> > Yeah, we don't tend to prepare the presentations until after our talks
> > have been accepted and that's when we use the publications repo.
> >
> > I like using the mailing list as a notification mechanism but I think
> > we should also maintain an etherpad with a listing so we can keep it
> > all organized in the longer term.
> >
> I'd be happy to see a etherpad URL we can use to list conferences and talks
> people are thinking of talking at.
>
> > --
> > Elizabeth Krumbach Joseph || Lyz || pleia2
> >
>
>
I like the coordination, sometimes it is useful if suddenly two folks are
interested in talking at the same conference (either can be a cooler bigger
talk, or each can be more specialized). :)

--Morgan
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Thirdparty CI: XenProject CI Failures result in 404 (no logs)

[Morgan Fainberg]

Hi Bob,

Thanks for looking into this.

Cheers,
--Morgan

On Wed, Dec 9, 2015 at 3:39 AM, Bob Ball <bob.b...@citrix.com> wrote:

> Sorry – just got to this mail.
>
>
>
> Just to be clear, this was a transient failure on the 7th December, which
> is now fixed.
>
>
>
> The most recent recheck was successful and
> http://zuul.openstack.xenproject.org/scoreboard/?project=openstack%2Fnova=jenkins%2CXenProject-CI=48==_size=100
> shows the XenProject CI to be healthy over the last 48 hours.
>
>
>
> Thanks,
>
>
>
> Bob
>
>
>
> *From:* Morgan Fainberg [mailto:morgan.fainb...@gmail.com]
> *Sent:* 08 December 2015 23:57
> *To:* openstack...@xenproject.org; OpenStack Infra
> *Subject:* Thirdparty CI: XenProject CI Failures result in 404 (no logs)
>
>
>
> As seen in changeset https://review.openstack.org/#/c/253792/ the 3rd
> party CI for XenProject CI check is resulting in HTTP 404 instead of useful
> logs. As this is a voting job, the lack of these logs make it hard to
> determine if the failure is related to the changeset or unrelated. The
> example failure is here:
> http://logs.openstack.xenproject.org/92/253792/7/check/dsvm-tempest-xen/1aa770f
> from the most recent failed run.
>
> I would like to ask that this is looked into so that we can continue to
> benefit from the XenProject CI reporting on patch-sets.
>
> Thanks!
>
> --Morgan
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] Thirdparty CI: XenProject CI Failures result in 404 (no logs)

[Morgan Fainberg]

As seen in changeset https://review.openstack.org/#/c/253792/ the 3rd party
CI for XenProject CI check is resulting in HTTP 404 instead of useful logs.
As this is a voting job, the lack of these logs make it hard to determine
if the failure is related to the changeset or unrelated. The example
failure is here:
http://logs.openstack.xenproject.org/92/253792/7/check/dsvm-tempest-xen/1aa770f
from the most recent failed run.

I would like to ask that this is looked into so that we can continue to
benefit from the XenProject CI reporting on patch-sets.

Thanks!
--Morgan
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra