Re: [openstack-dev] [tripleo][ci][infra] Quickstart Branching
On 5/23/18 6:49 PM, Sagi Shnaidman wrote:
Alex,
the problem is that you're working and focusing mostly on release
specific code like featuresets and some scripts. But
tripleo-quickstart(-extras) and tripleo-ci is much *much* more than set
of featuresets. Only 10% of the code may be related to releases and
branches, while other 90% is completely independent and not related to
releases.
So in 90% code we DO need to backport every change, take for example the
latest patch to extras: https://review.openstack.org/#/c/570167/, it's
fixing reproducer. If oooq-extra was branched, we would need to backport
this fix to every and every branch. And the same for all other 90% of
code, which is complete nonsense.
Just because not using "{% if release %}" construct - to block the whole
work of CI team and make the CI code is absolutely unmaintainable?
Some of release related templates we moved recently from tripleo-ci to
THT repo like scenarios, OC templates, etc. If we discover another
things in oooq that could be moved to branched THT I'd be only happy for
that.
Sometimes it could be hard to maintain one file in extras templates with
different logic for releases, like we have in tempest configuration for
example. The solution is to create a few release-related templates and
use one that match the current branch. It doesn't affect 90% of code and
still "branch-like" approach. But I didn't see other scripts that are so
release dependent. If we'll have ones, we could do the same. For now I
see "{% if release %}" construct working very well.
I didn't see still any advantage of branching CI code, except of a
little bit nicer jinja templates without "{% if release ", but amount of
disadvantages is so huge, that it'll literally block all current work in CI.
[tl;dr] branching allows to not run cloned branched jobs against master
patches. Or patches will wait longer in queues, and fail more often cuz
of intermittent infra issues. See explanation and some calculations below.
So my main concern against additional stable release cloned jobs
executed for master branches is that there is an "infra failure fee",
which is a failure unrelated to the patch under check or gate, like an
intermittent connectivity/timeout inducted failure. This normally is
followed by a 'recheck' comment posted by an engineer, and sometimes is
noticed by the elastic recheck bot as well. Say, that sort of a failure
has a probability of N. And the real "product failure", which is related
to the subject patch and not infra, takes P. So chances to fail for a job is
F = (1 - ((1 - N)*(1 - P)).
Now that we have added a two more "branched clones" for RDO CI OVB jobs
and a two more zuul jobs, we have this equation as
F = (1 - ((1 - N)^4*(1 - P)).
(I assumed the chances to face a product defect for the cloned branched
jobs remain unchanged).
This might bring significantly increased chances to fail (see some
examples [0] for the N/P distribution cases). So folks will start
posting 'recheck' comments now even more often, like x2 times more
often. Which would make zuul and RDO CI queues larger, and patches
sitting there longer - ending up with more time to wait for jobs to
start its check/gate pipelines. That's what I call 'recheck storms'. And
w/o branched quickstart/extras, we might have those storms amplified,
tho that fully depends on real N/P distributions.
[0] https://pastebin.com/ckG5G7NG
Thanks
On Wed, May 23, 2018 at 7:04 PM, Alex Schultz > wrote:
On Wed, May 23, 2018 at 8:30 AM, Sagi Shnaidman mailto:sshna...@redhat.com>> wrote:
> Hi, Sergii
>
> thanks for the question. It's not first time that this topic is raised and
> from first view it could seem that branching would help to that sort of
> issues.
>
> Although it's not the case. Tripleo-quickstart(-extras) is part of CI
code,
> as well as tripleo-ci repo which have never been branched. The reason for
> that is relative small impact on CI code from product branching. Think
about
> backport almost *every* patch to oooq and extras to all supported
branches,
> down to newton at least. This will be a really *huge* price and non
> reasonable work. Just think about active maintenance of 3-4 versions of CI
> code in each of 3 repositories. It will take all time of CI team with
almost
> zero value of this work.
>
So I'm not sure I completely agree with this assessment as there is a
price paid for every {%if release in [...]%} that we have to carry in
oooq{,-extras}. These go away if we branch because we don't have to
worry about breaking previous releases or current release (which may
or may not actually have CI results).
> What regards patch you listed, we would have backport this change to
*every*
> branch, and it wouldn't really help to avoid the issue. The source of
> problem is not branchless repo here.
>
Re: [openstack-dev] [tripleo][ci][infra] Quickstart Branching
Hi,
On Wed, May 23, 2018 at 8:20 PM, Sagi Shnaidman wrote:
>
>>
>> to reduce the impact of a change. From my original reply:
>>
>> > If there's a high maintenance cost, we haven't properly identified the
>> > optimal way to separate functionality between tripleo/quickstart.
>>
>> IMHO this is a side effect of having a whole bunch of roles in a
>> single repo. oooq-extras has a mix of tripleo and non-tripleo related
>> content. The reproducer IMHO is related to provisioning and could fall
>> in the oooq repo and not oooq-extras. This is a structure problem
>> with quickstart. If it's not version specific, then don't put it in a
>> version specific repo. But that doesn't mean don't use version
>> specific repos at all.
>>
>> This is one of the reasons why we're opting not to use this pattern of
>> a bunch of roles in a single repo for tripleo itself[0][1][2]. We
>> learned with the puppet modules that carrying all this stuff in a
>> single repo has a huge maintenance cost and if you split them out you
>> can identify re-usability and establish proper patterns for moving
>> functionality into a shared place[3]. Yes there is a maintenance cost
>> of maintaining independent repos, but at the same time there's a
>> benefit of re-usability by other projects/groups when you expose
>> important pieces of functionality as a standalone. You can establish
>> clear ways to interact with each piece, test items, and release
>> independently. For example the ansible-role-container-registry is not
>> tripleo specific and anyone looking to manage a standalone docker
>> registry can use it & contribute.
>>
>
> We were moving between having all roles in one repo and having a separate
> repo for each role a few times. Each case has it's advantages and
> disadvantages. Last time we moved to have roles in 2 repos - quickstart and
> extras, it was a year ago I think. So far IMHO it's the best approach. There
> will be a mechanism to install additional roles, like we have for
> tirpleo-upgrade, ops-tools, etc etc.
But at the moment we don't have that mechanism so we should live
somehow until it's implemented.
> It may be a much broader topic to discuss, although I think having part of
> roles branched and part of not branched is much more headache.
> Tripleo-upgrade is a good example of it.
>
>>
>> > So in 90% code we DO need to backport every change, take for example the
>> > latest patch to extras: https://review.openstack.org/#/c/570167/, it's
>> > fixing reproducer. If oooq-extra was branched, we would need to backport
>> > this fix to every and every branch. And the same for all other 90% of
>> > code,
>> > which is complete nonsense.
>> > Just because not using "{% if release %}" construct - to block the whole
>> > work of CI team and make the CI code is absolutely unmaintainable?
>> >
>>
>> And you're saying what we currently have is maintainable? We keep
>> breaking ourselves, there's big gaps in coverage and it takes
>> time[4][5] to identify breakages. I don't consider that maintainable
>> because this is a recurring topic because we clearly haven't fixed it
>> with the current setup. It's time to re-evaluate what we have an see
>> if there's room for improvement. I know I wasn't proposing to branch
>> all the repositories, but it might make sense to figure out if there's
>> a way to reduce our recurring issues with stable branches or
>> independent modules for some of the functions in CI.
>
>
>> Considering this is how we broke Queens, I'm not sure I agree.
We broke Queens, Pike, Newton by merging [1] without testing against
these releases.
>>
>
> First of all I don't see any connection between maintenance and CI
> breakages, it's different topics. And yes, it IS maintainable CI that we
> have now, and I have what to compare it with. I remember very well
> tripleo.sh based approach, also you can see almost green dashboards last
> time which proves my statement. CI is not ideal now, but it's definitely
> much better than 1-2 years ago.
>
>
> Of course we have breakages, the CI is actually history of breakages and
> fixes, as any other product. Wrt queens issue, it took about a week to solve
> it not because it was so hard, but because we had a very difficult weeks
> when trying to fix all Centos 7.5 issues and queens branch was in second
> priority. And by the way, we fixed everything much faster then it was with
> CentOS 7.4. Having the negative attitude that every CI breakage is proof of
> wrong CI structure is not correct and doesn't help. If branching helped in
> this case, it would create much bigger problems in all other cases.
I would like to forget about feeling and discuss the technical side of
2 solutions, cost for every team and product in general to find the
solution that fits all.
>
> Anyway, we saw that having branch jobs in OVB only didn't catch queens issue
> (why - you know better) so we added multinode branch specific ones, which
> will catch such issues in the future. We hit the probl
Re: [openstack-dev] [tripleo][ci][infra] Quickstart Branching
> to reduce the impact of a change. From my original reply:
>
> > If there's a high maintenance cost, we haven't properly identified the
> optimal way to separate functionality between tripleo/quickstart.
>
> IMHO this is a side effect of having a whole bunch of roles in a
> single repo. oooq-extras has a mix of tripleo and non-tripleo related
> content. The reproducer IMHO is related to provisioning and could fall
> in the oooq repo and not oooq-extras. This is a structure problem
> with quickstart. If it's not version specific, then don't put it in a
> version specific repo. But that doesn't mean don't use version
> specific repos at all.
>
> This is one of the reasons why we're opting not to use this pattern of
> a bunch of roles in a single repo for tripleo itself[0][1][2]. We
> learned with the puppet modules that carrying all this stuff in a
> single repo has a huge maintenance cost and if you split them out you
> can identify re-usability and establish proper patterns for moving
> functionality into a shared place[3]. Yes there is a maintenance cost
> of maintaining independent repos, but at the same time there's a
> benefit of re-usability by other projects/groups when you expose
> important pieces of functionality as a standalone. You can establish
> clear ways to interact with each piece, test items, and release
> independently. For example the ansible-role-container-registry is not
> tripleo specific and anyone looking to manage a standalone docker
> registry can use it & contribute.
>
>
We were moving between having all roles in one repo and having a separate
repo for each role a few times. Each case has it's advantages and
disadvantages. Last time we moved to have roles in 2 repos - quickstart and
extras, it was a year ago I think. So far IMHO it's the best approach.
There will be a mechanism to install additional roles, like we have for
tirpleo-upgrade, ops-tools, etc etc.
It may be a much broader topic to discuss, although I think having part of
roles branched and part of not branched is much more headache.
Tripleo-upgrade is a good example of it.
> > So in 90% code we DO need to backport every change, take for example the
> > latest patch to extras: https://review.openstack.org/#/c/570167/, it's
> > fixing reproducer. If oooq-extra was branched, we would need to backport
> > this fix to every and every branch. And the same for all other 90% of
> code,
> > which is complete nonsense.
> > Just because not using "{% if release %}" construct - to block the whole
> > work of CI team and make the CI code is absolutely unmaintainable?
> >
>
> And you're saying what we currently have is maintainable? We keep
> breaking ourselves, there's big gaps in coverage and it takes
> time[4][5] to identify breakages. I don't consider that maintainable
> because this is a recurring topic because we clearly haven't fixed it
> with the current setup. It's time to re-evaluate what we have an see
> if there's room for improvement. I know I wasn't proposing to branch
> all the repositories, but it might make sense to figure out if there's
> a way to reduce our recurring issues with stable branches or
> independent modules for some of the functions in CI.
>
Considering this is how we broke Queens, I'm not sure I agree.
>
>
First of all I don't see any connection between maintenance and CI
breakages, it's different topics. And yes, it IS maintainable CI that we
have now, and I have what to compare it with. I remember very well
tripleo.sh based approach, also you can see almost green dashboards last
time which proves my statement. CI is not ideal now, but it's definitely
much better than 1-2 years ago.
Of course we have breakages, the CI is actually history of breakages and
fixes, as any other product. Wrt queens issue, it took about a week to
solve it not because it was so hard, but because we had a very difficult
weeks when trying to fix all Centos 7.5 issues and queens branch was in
second priority. And by the way, we fixed everything much faster then it
was with CentOS 7.4. Having the negative attitude that every CI breakage
is proof of wrong CI structure is not correct and doesn't help. If
branching helped in this case, it would create much bigger problems in all
other cases.
Anyway, we saw that having branch jobs in OVB only didn't catch queens
issue (why - you know better) so we added multinode branch specific ones,
which will catch such issues in the future. We hit the problem, solved it,
set preventive actions and are ready to catch it next time. This is a
normal CI workflow and I don't see any problem with it. Having multinode
branch jobs is actually pretty similar to "branching" repos but without
maintenance nightmare.
Thanks
Thanks,
> -Alex
>
> [0] http://git.openstack.org/cgit/openstack/ansible-role-
> container-registry/
> [1] http://git.openstack.org/cgit/openstack/ansible-role-redhat-
> subscription/
> [2] http://git.openstack.org/cgit/openstack/ansible-role-tripleo-keystone/
> [3] htt
Re: [openstack-dev] [tripleo][ci][infra] Quickstart Branching
On Wed, May 23, 2018 at 10:49 AM, Sagi Shnaidman wrote:
> Alex,
>
> the problem is that you're working and focusing mostly on release specific
> code like featuresets and some scripts. But tripleo-quickstart(-extras) and
> tripleo-ci is much *much* more than set of featuresets. Only 10% of the code
> may be related to releases and branches, while other 90% is completely
> independent and not related to releases.
>
It is not necessarily about release specific code, it's about being
able to reduce the impact of a change. From my original reply:
> If there's a high maintenance cost, we haven't properly identified the
> optimal way to separate functionality between tripleo/quickstart.
IMHO this is a side effect of having a whole bunch of roles in a
single repo. oooq-extras has a mix of tripleo and non-tripleo related
content. The reproducer IMHO is related to provisioning and could fall
in the oooq repo and not oooq-extras. This is a structure problem
with quickstart. If it's not version specific, then don't put it in a
version specific repo. But that doesn't mean don't use version
specific repos at all.
This is one of the reasons why we're opting not to use this pattern of
a bunch of roles in a single repo for tripleo itself[0][1][2]. We
learned with the puppet modules that carrying all this stuff in a
single repo has a huge maintenance cost and if you split them out you
can identify re-usability and establish proper patterns for moving
functionality into a shared place[3]. Yes there is a maintenance cost
of maintaining independent repos, but at the same time there's a
benefit of re-usability by other projects/groups when you expose
important pieces of functionality as a standalone. You can establish
clear ways to interact with each piece, test items, and release
independently. For example the ansible-role-container-registry is not
tripleo specific and anyone looking to manage a standalone docker
registry can use it & contribute.
> So in 90% code we DO need to backport every change, take for example the
> latest patch to extras: https://review.openstack.org/#/c/570167/, it's
> fixing reproducer. If oooq-extra was branched, we would need to backport
> this fix to every and every branch. And the same for all other 90% of code,
> which is complete nonsense.
> Just because not using "{% if release %}" construct - to block the whole
> work of CI team and make the CI code is absolutely unmaintainable?
>
And you're saying what we currently have is maintainable? We keep
breaking ourselves, there's big gaps in coverage and it takes
time[4][5] to identify breakages. I don't consider that maintainable
because this is a recurring topic because we clearly haven't fixed it
with the current setup. It's time to re-evaluate what we have an see
if there's room for improvement. I know I wasn't proposing to branch
all the repositories, but it might make sense to figure out if there's
a way to reduce our recurring issues with stable branches or
independent modules for some of the functions in CI.
> Some of release related templates we moved recently from tripleo-ci to THT
> repo like scenarios, OC templates, etc. If we discover another things in
> oooq that could be moved to branched THT I'd be only happy for that.
>
> Sometimes it could be hard to maintain one file in extras templates with
> different logic for releases, like we have in tempest configuration for
> example. The solution is to create a few release-related templates and use
> one that match the current branch. It doesn't affect 90% of code and still
> "branch-like" approach. But I didn't see other scripts that are so release
> dependent. If we'll have ones, we could do the same. For now I see "{% if
> release %}" construct working very well.
Considering this is how we broke Queens, I'm not sure I agree.
>
> I didn't see still any advantage of branching CI code, except of a little
> bit nicer jinja templates without "{% if release ", but amount of
> disadvantages is so huge, that it'll literally block all current work in CI.
>
It's about reducing our risk with test coverage. We do not properly
test all jobs and all configurations when we make these changes. This
is a repeated problem and when we have to add version specific logic,
unless we're able to identify what this is actually impacting and
verify with jobs we have a risk of breaking ourselves. We've seen
that code review is not sufficient for these changes as we merge
things and only find out after they've been merged that we broke
stable branches. Then it takes folks tracking down changes to decipher
what we broke. For example the original patch[4] broke Queens for
about a week. That's 7 days of nothing being able to be merged,
that's not OK.
Thanks,
-Alex
[0] http://git.openstack.org/cgit/openstack/ansible-role-container-registry/
[1] http://git.openstack.org/cgit/openstack/ansible-role-redhat-subscription/
[2] http://git.openstack.org/cgit/openstack/ansible-role-tripleo-keystone/
[3
Re: [openstack-dev] [tripleo][ci][infra] Quickstart Branching
Alex,
the problem is that you're working and focusing mostly on release specific
code like featuresets and some scripts. But tripleo-quickstart(-extras) and
tripleo-ci is much *much* more than set of featuresets. Only 10% of the
code may be related to releases and branches, while other 90% is completely
independent and not related to releases.
So in 90% code we DO need to backport every change, take for example the
latest patch to extras: https://review.openstack.org/#/c/570167/, it's
fixing reproducer. If oooq-extra was branched, we would need to backport
this fix to every and every branch. And the same for all other 90% of code,
which is complete nonsense.
Just because not using "{% if release %}" construct - to block the whole
work of CI team and make the CI code is absolutely unmaintainable?
Some of release related templates we moved recently from tripleo-ci to THT
repo like scenarios, OC templates, etc. If we discover another things in
oooq that could be moved to branched THT I'd be only happy for that.
Sometimes it could be hard to maintain one file in extras templates with
different logic for releases, like we have in tempest configuration for
example. The solution is to create a few release-related templates and use
one that match the current branch. It doesn't affect 90% of code and still
"branch-like" approach. But I didn't see other scripts that are so release
dependent. If we'll have ones, we could do the same. For now I see "{% if
release %}" construct working very well.
I didn't see still any advantage of branching CI code, except of a little
bit nicer jinja templates without "{% if release ", but amount of
disadvantages is so huge, that it'll literally block all current work in CI.
Thanks
On Wed, May 23, 2018 at 7:04 PM, Alex Schultz wrote:
> On Wed, May 23, 2018 at 8:30 AM, Sagi Shnaidman
> wrote:
> > Hi, Sergii
> >
> > thanks for the question. It's not first time that this topic is raised
> and
> > from first view it could seem that branching would help to that sort of
> > issues.
> >
> > Although it's not the case. Tripleo-quickstart(-extras) is part of CI
> code,
> > as well as tripleo-ci repo which have never been branched. The reason for
> > that is relative small impact on CI code from product branching. Think
> about
> > backport almost *every* patch to oooq and extras to all supported
> branches,
> > down to newton at least. This will be a really *huge* price and non
> > reasonable work. Just think about active maintenance of 3-4 versions of
> CI
> > code in each of 3 repositories. It will take all time of CI team with
> almost
> > zero value of this work.
> >
>
> So I'm not sure I completely agree with this assessment as there is a
> price paid for every {%if release in [...]%} that we have to carry in
> oooq{,-extras}. These go away if we branch because we don't have to
> worry about breaking previous releases or current release (which may
> or may not actually have CI results).
>
> > What regards patch you listed, we would have backport this change to
> *every*
> > branch, and it wouldn't really help to avoid the issue. The source of
> > problem is not branchless repo here.
> >
>
> No we shouldn't be backporting every change. The logic in oooq-extras
> should be version specific and if we're changing an interface in
> tripleo in a breaking fashion we're doing it wrong in tripleo. If
> we're backporting things to work around tripleo issues, we're doing it
> wrong in quickstart.
>
> > Regarding catching such issues and Bogdans point, that's right we added a
> > few jobs to catch such issues in the future and prevent breakages, and a
> few
> > running jobs is reasonable price to keep configuration working in all
> > branches. Comparing to maintenance nightmare with branches of CI code,
> it's
> > really a *zero* price.
> >
>
> Nothing is free. If there's a high maintenance cost, we haven't
> properly identified the optimal way to separate functionality between
> tripleo/quickstart. I have repeatedly said that the provisioning
> parts of quickstart should be separate because those aren't tied to a
> tripleo version and this along with the scenario configs should be the
> only unbranched repo we have. Any roles related to how to
> configure/work with tripleo should be branched and tied to a stable
> branch of tripleo. This would actually be beneficial for tripleo as
> well because then we can see when we are introducing backwards
> incompatible changes.
>
> Thanks,
> -Alex
>
> > Thanks
> >
> >
> > On Wed, May 23, 2018 at 3:43 PM, Sergii Golovatiuk
> > wrote:
> >>
> >> Hi,
> >>
> >> Looking at [1], I am thinking about the price we paid for not
> >> branching tripleo-quickstart. Can we discuss the options to prevent
> >> the issues such as [1]? Thank you in advance.
> >>
> >> [1] https://review.openstack.org/#/c/569830/4
> >>
> >> --
> >> Best Regards,
> >> Sergii Golovatiuk
> >>
> >>
> __
> >> Open
Re: [openstack-dev] [tripleo][ci][infra] Quickstart Branching
On Wed, May 23, 2018 at 8:30 AM, Sagi Shnaidman wrote:
> Hi, Sergii
>
> thanks for the question. It's not first time that this topic is raised and
> from first view it could seem that branching would help to that sort of
> issues.
>
> Although it's not the case. Tripleo-quickstart(-extras) is part of CI code,
> as well as tripleo-ci repo which have never been branched. The reason for
> that is relative small impact on CI code from product branching. Think about
> backport almost *every* patch to oooq and extras to all supported branches,
> down to newton at least. This will be a really *huge* price and non
> reasonable work. Just think about active maintenance of 3-4 versions of CI
> code in each of 3 repositories. It will take all time of CI team with almost
> zero value of this work.
>
So I'm not sure I completely agree with this assessment as there is a
price paid for every {%if release in [...]%} that we have to carry in
oooq{,-extras}. These go away if we branch because we don't have to
worry about breaking previous releases or current release (which may
or may not actually have CI results).
> What regards patch you listed, we would have backport this change to *every*
> branch, and it wouldn't really help to avoid the issue. The source of
> problem is not branchless repo here.
>
No we shouldn't be backporting every change. The logic in oooq-extras
should be version specific and if we're changing an interface in
tripleo in a breaking fashion we're doing it wrong in tripleo. If
we're backporting things to work around tripleo issues, we're doing it
wrong in quickstart.
> Regarding catching such issues and Bogdans point, that's right we added a
> few jobs to catch such issues in the future and prevent breakages, and a few
> running jobs is reasonable price to keep configuration working in all
> branches. Comparing to maintenance nightmare with branches of CI code, it's
> really a *zero* price.
>
Nothing is free. If there's a high maintenance cost, we haven't
properly identified the optimal way to separate functionality between
tripleo/quickstart. I have repeatedly said that the provisioning
parts of quickstart should be separate because those aren't tied to a
tripleo version and this along with the scenario configs should be the
only unbranched repo we have. Any roles related to how to
configure/work with tripleo should be branched and tied to a stable
branch of tripleo. This would actually be beneficial for tripleo as
well because then we can see when we are introducing backwards
incompatible changes.
Thanks,
-Alex
> Thanks
>
>
> On Wed, May 23, 2018 at 3:43 PM, Sergii Golovatiuk
> wrote:
>>
>> Hi,
>>
>> Looking at [1], I am thinking about the price we paid for not
>> branching tripleo-quickstart. Can we discuss the options to prevent
>> the issues such as [1]? Thank you in advance.
>>
>> [1] https://review.openstack.org/#/c/569830/4
>>
>> --
>> Best Regards,
>> Sergii Golovatiuk
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
> Best regards
> Sagi Shnaidman
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo][ci][infra] Quickstart Branching
Hi, Sergii thanks for the question. It's not first time that this topic is raised and from first view it could seem that branching would help to that sort of issues. Although it's not the case. Tripleo-quickstart(-extras) is part of CI code, as well as tripleo-ci repo which have never been branched. The reason for that is relative small impact on CI code from product branching. Think about backport almost *every* patch to oooq and extras to all supported branches, down to newton at least. This will be a really *huge* price and non reasonable work. Just think about active maintenance of 3-4 versions of CI code in each of 3 repositories. It will take all time of CI team with almost zero value of this work. What regards patch you listed, we would have backport this change to *every* branch, and it wouldn't really help to avoid the issue. The source of problem is not branchless repo here. Regarding catching such issues and Bogdans point, that's right we added a few jobs to catch such issues in the future and prevent breakages, and a few running jobs is reasonable price to keep configuration working in all branches. Comparing to maintenance nightmare with branches of CI code, it's really a *zero* price. Thanks On Wed, May 23, 2018 at 3:43 PM, Sergii Golovatiuk wrote: > Hi, > > Looking at [1], I am thinking about the price we paid for not > branching tripleo-quickstart. Can we discuss the options to prevent > the issues such as [1]? Thank you in advance. > > [1] https://review.openstack.org/#/c/569830/4 > > -- > Best Regards, > Sergii Golovatiuk > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards Sagi Shnaidman __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tripleo][ci][infra] Quickstart Branching
On 5/23/18 2:43 PM, Sergii Golovatiuk wrote: Hi, Looking at [1], I am thinking about the price we paid for not branching tripleo-quickstart. Can we discuss the options to prevent the issues such as [1]? Thank you in advance. [1] https://review.openstack.org/#/c/569830/4 That was only a half of the full price, actually, see also additional multinode containers check/gate jobs [0],[1] from now on executed against the master branches of all tripleo repos (IIUC), for release -2 and -1 from master. [0] https://review.openstack.org/#/c/569932/ [1] https://review.openstack.org/#/c/569854/ -- Best regards, Bogdan Dobrelya, Irc #bogdando __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tripleo][ci][infra] Quickstart Branching
Hi, Looking at [1], I am thinking about the price we paid for not branching tripleo-quickstart. Can we discuss the options to prevent the issues such as [1]? Thank you in advance. [1] https://review.openstack.org/#/c/569830/4 -- Best Regards, Sergii Golovatiuk __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
