Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

[Maciej Kwiek]

Artem,

[1] and [2] are changes that do exactly this - snapshot is created in
/var/log. Please review these changes if you haven't already.

Cheers,
Maciej

[1] https://review.openstack.org/#/c/270823/
[2] https://review.openstack.org/#/c/271179/

On Mon, Jan 25, 2016 at 5:09 PM, Artem Panchenko <apanche...@mirantis.com>
wrote:

> Guys,
>
> I want to pay your attention that we need to not only fix snapshots
> generation issue, but also prevent caused by it unexpected services
> failures (see details in a duplicate [0] of original [1] bug), which would
> become a challenge for not experienced users (for example he/she won't be
> able to authenticate in GUI or CLI some time after snapshot generation is
> started). I get this issue on our bare-metal lab (10 slaves) each time I
> have an environment which is running more than 2 days.
>
> Links usage for files copying doesn't help in such case, because tarball
> is still saved on /var partition. Also, if I want to workaround this issue,
> I have to perform a lot of actions: s hrink 'os-varlog' volume (because
> it's the biggest [2] one) in order to increase unallocated disk space,
> resize its FS, create new volume, create FS, mount it to
> /var/www/nailgun/dump and update fstab. Not easy way to make "Generate
> Diagnostic Snapshot " button work, right?
>
> So, if we are going to address this diagnostic snapshot issue in 8.0, I
> want to remind you about b) option :)
>
> b) Make the snapshot location share the diskspace of /var/log?
>
>
> Thanks!
>
> [0] https://bugs.launchpad.net/fuel/+bug/1530131
> [1] https://bugs.launchpad.net/fuel/+bug/1529182
> [2] http://paste.openstack.org/show/484895/
>
>
> On 18.01.16 13:20, Maciej Kwiek wrote:
>
> Igor: It seems that fqdn -> ipaddr will indeed be resolved. Please share
> your feedback in review:  <https://review.openstack.org/#/c/266964/3>
> https://review.openstack.org/#/c/266964/3
>
> On Fri, Jan 15, 2016 at 4:25 PM, Igor Kalnitsky <ikalnit...@mirantis.com>
> wrote:
>
>> Sheena -
>>
>> What do you mean by *targeted*? Shotgun's designed to be a *targeted*
>> solution. If someone wants more *precise* targets - it's easy to
>> specify them in Nailgun's settings.yaml.
>>
>> - Igor
>>
>> On Fri, Jan 15, 2016 at 5:02 PM, Sheena Gregson < <sgreg...@mirantis.com>
>> sgreg...@mirantis.com> wrote:
>> > I’ve also seen the request multiple times to be able to provide more
>> > targeted snapshots which might also (partially) solve this problem as it
>> > would require significantly less disk space to grab logs from a subset
>> of
>> > nodes for a specific window of time, instead of the more robust grab-all
>> > solution we have now.
>> >
>> >
>> >
>> > From: Maciej Kwiek [mailto: <mkw...@mirantis.com>mkw...@mirantis.com]
>> > Sent: Thursday, January 14, 2016 5:59 AM
>> > To: OpenStack Development Mailing List (not for usage questions)
>> > <openstack-dev@lists.openstack.org>
>> > Subject: Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is
>> broken
>> > due to lack of disk space
>> >
>> >
>> >
>> > Igor,
>> >
>> >
>> >
>> > I will investigate this, thanks!
>> >
>> >
>> >
>> > Artem,
>> >
>> >
>> >
>> > I guess that if we have an untrusted user on master node, he could just
>> put
>> > something he wants to be in the snapshot in /var/log without having to
>> time
>> > the attack carefully with tar execution.
>> >
>> >
>> >
>> > I want to use links for directories, this saves me the trouble of
>> creating
>> > hardlinks for every single file in the directory. Although with how
>> > exclusion is currently implemented it can cause deleting log files from
>> > original directories, need to check this out.
>> >
>> >
>> >
>> > About your PS: whole /var/log on master node (not in container) is
>> currently
>> > downloaded, I think we shouldn't change this as we plan to drop
>> containers
>> > in 9.0.
>> >
>> >
>> >
>> > Cheers,
>> >
>> > Maciej
>> >
>> >
>> >
>> > On Thu, Jan 14, 2016 at 12:32 PM, Artem Panchenko <
>> apanche...@mirantis.com>
>> > wrote:
>> >
>> > Hi,
>> >
>> > using symlinks is a bit dangerous, here is a quote from the man you
>> > mentioned [0]:
>> >
>> >> T

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

[Maciej Kwiek]

Igor: It seems that fqdn -> ipaddr will indeed be resolved. Please share
your feedback in review: https://review.openstack.org/#/c/266964/3

On Fri, Jan 15, 2016 at 4:25 PM, Igor Kalnitsky <ikalnit...@mirantis.com>
wrote:

> Sheena -
>
> What do you mean by *targeted*? Shotgun's designed to be a *targeted*
> solution. If someone wants more *precise* targets - it's easy to
> specify them in Nailgun's settings.yaml.
>
> - Igor
>
> On Fri, Jan 15, 2016 at 5:02 PM, Sheena Gregson <sgreg...@mirantis.com>
> wrote:
> > I’ve also seen the request multiple times to be able to provide more
> > targeted snapshots which might also (partially) solve this problem as it
> > would require significantly less disk space to grab logs from a subset of
> > nodes for a specific window of time, instead of the more robust grab-all
> > solution we have now.
> >
> >
> >
> > From: Maciej Kwiek [mailto:mkw...@mirantis.com]
> > Sent: Thursday, January 14, 2016 5:59 AM
> > To: OpenStack Development Mailing List (not for usage questions)
> > <openstack-dev@lists.openstack.org>
> > Subject: Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is
> broken
> > due to lack of disk space
> >
> >
> >
> > Igor,
> >
> >
> >
> > I will investigate this, thanks!
> >
> >
> >
> > Artem,
> >
> >
> >
> > I guess that if we have an untrusted user on master node, he could just
> put
> > something he wants to be in the snapshot in /var/log without having to
> time
> > the attack carefully with tar execution.
> >
> >
> >
> > I want to use links for directories, this saves me the trouble of
> creating
> > hardlinks for every single file in the directory. Although with how
> > exclusion is currently implemented it can cause deleting log files from
> > original directories, need to check this out.
> >
> >
> >
> > About your PS: whole /var/log on master node (not in container) is
> currently
> > downloaded, I think we shouldn't change this as we plan to drop
> containers
> > in 9.0.
> >
> >
> >
> > Cheers,
> >
> > Maciej
> >
> >
> >
> > On Thu, Jan 14, 2016 at 12:32 PM, Artem Panchenko <
> apanche...@mirantis.com>
> > wrote:
> >
> > Hi,
> >
> > using symlinks is a bit dangerous, here is a quote from the man you
> > mentioned [0]:
> >
> >> The `--dereference' option is unsafe if an untrusted user can modify
> >> directories while tar is running.
> >
> > Hard links usage is much safer, because you can't use them for
> directories.
> > But at the same time implementation in shotgun would be more complicated
> > than with symlinks.
> >
> > Anyway, in order to determine what linking to use we need to decide where
> > (/var/log or another partition) diagnostic snapshot will be stored.
> >
> > p.s.
> >
> >>This doesn't really give us much right now, because most of the logs are
> >> fetched from master node via ssh due to shotgun being run in mcollective
> >> container
> >
> >
> >
> > AFAIK '/var/log/docker-logs/' is available from mcollective container and
> > mounted to /var/log/:
> >
> > [root@fuel-lab-cz5557 ~]# dockerctl shell mcollective mount -l | grep
> > os-varlog
> > /dev/mapper/os-varlog on /var/log type ext4
> > (rw,relatime,stripe=128,data=ordered)
> >
> > From my experience '/var/log/docker-logs/remote' folder is most ' heavy'
> > thing in snapshot.
> >
> > [0] http://www.gnu.org/software/tar/manual/html_node/dereference.html
> >
> > Thanks!
> >
> >
> >
> > On 14.01.16 13:00, Igor Kalnitsky wrote:
> >
> > I took a glance on Maciej's patch and it adds a switch to tar command
> >
> > to make it follow symbolic links
> >
> > Yeah, that should work. Except one thing - we previously had fqdn ->
> >
> > ipaddr links in snapshots. So now they will be resolved into full
> >
> > copy?
> >
> >
> >
> > I meant that symlinks also give us the benefit of not using additional
> >
> > space (just as hardlinks do) while being able to link to files from
> >
> > different filesystems.
> >
> > I'm sorry, I got you wrong. :)
> >
> >
> >
> > - Igor
> >
> >
> >
> > On Thu, Jan 14, 2016 at 12:34 PM, Maciej Kwiek <mkw...@mirantis.com>
> wrote:
> >
> > Igor,
> >
> >
> >
> > I meant

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

[Maciej Kwiek]

Thanks for your insight guys!

I agree with Oleg, I will see what I can do to make this work this way.

About hardlinks - wouldn't it be better to use symlinks? This way we don't
occupy more space than necessary, and we can link to files and directories
that are in other block device than /var. Please see [1] review for a
proposed change that introduces symlinks.

This doesn't really give us much right now, because most of the logs are
fetched from master node via ssh due to shotgun being run in mcollective
container, but it's something! When we remove containers, this will prove
more useful.

Regards,
Maciej Kwiek

[1] https://review.openstack.org/#/c/266964/

On Tue, Jan 12, 2016 at 1:51 PM, Oleg Gelbukh <ogelb...@mirantis.com> wrote:

> I think we need to find a way to:
>
> 1) verify the size of snapshot without actually making it and compare to
> the available disk space beforehand.
> 2) refuse to create snapshot if space is insufficient and notify user
> (otherwise it breaks Admin node as we have seen)
> 3) provide a way to prioritize elements of the snapshot and exclude them
> based on the priorities or user choice.
>
> This will allow for better and safer UX with the snapshot.
>
> --
> Best regards,
> Oleg Gelbukh
>
> On Tue, Jan 12, 2016 at 1:47 PM, Maciej Kwiek <mkw...@mirantis.com> wrote:
>
>> Hi!
>>
>> I need some advice on how to tackle this issue. There is a bug [1]
>> describing the problem with creating a diagnostic snapshot. The issue is
>> that /var/log has 100GB available, while /var (where diagnostic snapshot is
>> being generated - /var/www/nailgun/dump/fuel-snapshot according to [2]) has
>> 10GB available, so dumping the logs can be an issue when logs size exceed
>> free space in /var.
>>
>> There are several things we could do, but I am unsure on which course to
>> take. Should we
>> a) Allocate more disk space for /var/www (or for whole /var)?
>> b) Make the snapshot location share the diskspace of /var/log?
>> c) Something else? What?
>>
>> Please share your thoughts on this.
>>
>> Cheers,
>> Maciej Kwiek
>>
>> [1] https://bugs.launchpad.net/fuel/+bug/1529182
>> [2]
>> https://github.com/openstack/fuel-web/blob/2855a9ba925c146b4802ab3cd2185f1dce2d8a6a/nailgun/nailgun/settings.yaml#L717
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

[Maciej Kwiek]

Igor,

I meant that symlinks also give us the benefit of not using additional
space (just as hardlinks do) while being able to link to files from
different filesystems.

Also, as Barłomiej pointed out the `h` switch for tar should do the trick
[1].

Cheers,
Maciej

[1] http://www.gnu.org/software/tar/manual/html_node/dereference.html

On Thu, Jan 14, 2016 at 11:22 AM, Bartlomiej Piotrowski <
bpiotrow...@mirantis.com> wrote:

> Igor,
>
> I took a glance on Maciej's patch and it adds a switch to tar command to
> make it follow symbolic links, so it looks good to me.
>
> Bartłomiej
>
> On Thu, Jan 14, 2016 at 10:39 AM, Igor Kalnitsky <ikalnit...@mirantis.com>
> wrote:
>
>> Hey Maceij -
>>
>> > About hardlinks - wouldn't it be better to use symlinks?
>> > This way we don't occupy more space than necessary
>>
>> AFAIK, hardlinks won't occupy much space. They are the links, after all.
>> :)
>>
>> As for symlinks, I'm afraid shotgun (and fabric underneath) won't
>> resolve them and links are get to snapshot As Is. That means if there
>> will be no content in the snapshot they are pointing to, they are
>> simply useless. Needs to be checked, though.
>>
>> - Igor
>>
>> On Thu, Jan 14, 2016 at 10:31 AM, Maciej Kwiek <mkw...@mirantis.com>
>> wrote:
>> > Thanks for your insight guys!
>> >
>> > I agree with Oleg, I will see what I can do to make this work this way.
>> >
>> > About hardlinks - wouldn't it be better to use symlinks? This way we
>> don't
>> > occupy more space than necessary, and we can link to files and
>> directories
>> > that are in other block device than /var. Please see [1] review for a
>> > proposed change that introduces symlinks.
>> >
>> > This doesn't really give us much right now, because most of the logs are
>> > fetched from master node via ssh due to shotgun being run in mcollective
>> > container, but it's something! When we remove containers, this will
>> prove
>> > more useful.
>> >
>> > Regards,
>> > Maciej Kwiek
>> >
>> > [1] https://review.openstack.org/#/c/266964/
>> >
>> > On Tue, Jan 12, 2016 at 1:51 PM, Oleg Gelbukh <ogelb...@mirantis.com>
>> wrote:
>> >>
>> >> I think we need to find a way to:
>> >>
>> >> 1) verify the size of snapshot without actually making it and compare
>> to
>> >> the available disk space beforehand.
>> >> 2) refuse to create snapshot if space is insufficient and notify user
>> >> (otherwise it breaks Admin node as we have seen)
>> >> 3) provide a way to prioritize elements of the snapshot and exclude
>> them
>> >> based on the priorities or user choice.
>> >>
>> >> This will allow for better and safer UX with the snapshot.
>> >>
>> >> --
>> >> Best regards,
>> >> Oleg Gelbukh
>> >>
>> >> On Tue, Jan 12, 2016 at 1:47 PM, Maciej Kwiek <mkw...@mirantis.com>
>> wrote:
>> >>>
>> >>> Hi!
>> >>>
>> >>> I need some advice on how to tackle this issue. There is a bug [1]
>> >>> describing the problem with creating a diagnostic snapshot. The issue
>> is
>> >>> that /var/log has 100GB available, while /var (where diagnostic
>> snapshot is
>> >>> being generated - /var/www/nailgun/dump/fuel-snapshot according to
>> [2]) has
>> >>> 10GB available, so dumping the logs can be an issue when logs size
>> exceed
>> >>> free space in /var.
>> >>>
>> >>> There are several things we could do, but I am unsure on which course
>> to
>> >>> take. Should we
>> >>> a) Allocate more disk space for /var/www (or for whole /var)?
>> >>> b) Make the snapshot location share the diskspace of /var/log?
>> >>> c) Something else? What?
>> >>>
>> >>> Please share your thoughts on this.
>> >>>
>> >>> Cheers,
>> >>> Maciej Kwiek
>> >>>
>> >>> [1] https://bugs.launchpad.net/fuel/+bug/1529182
>> >>> [2]
>> >>>
>> https://github.com/openstack/fuel-web/blob/2855a9ba925c146b4802ab3cd2185f1dce2d8a6a/nailgun/nailgun/settings.yaml#L717
>> >>>
>> >>>
>> >>>
>> __
>> >>> OpenStack Development 

Re: [openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

[Maciej Kwiek]

Igor,

I will investigate this, thanks!

Artem,

I guess that if we have an untrusted user on master node, he could just put
something he wants to be in the snapshot in /var/log without having to time
the attack carefully with tar execution.

I want to use links for directories, this saves me the trouble of creating
hardlinks for every single file in the directory. Although with how
exclusion is currently implemented it can cause deleting log files from
original directories, need to check this out.

About your PS: whole /var/log on master node (not in container) is
currently downloaded, I think we shouldn't change this as we plan to drop
containers in 9.0.

Cheers,
Maciej

On Thu, Jan 14, 2016 at 12:32 PM, Artem Panchenko <apanche...@mirantis.com>
wrote:

> Hi,
>
> using symlinks is a bit dangerous, here is a quote from the man you
> mentioned [0]:
>
> > The `--dereference' option is unsafe if an untrusted user can modify
> directories while tar is running.
>
> Hard links usage is much safer, because you can't use them for
> directories. But at the same time implementation in shotgun would be more
> complicated than with symlinks.
>
> Anyway, in order to determine what linking to use we need to decide where
> (/var/log or another partition) diagnostic snapshot will be stored.
>
> p.s.
>
> >This doesn't really give us much right now, because most of the logs are 
> >fetched from master node via ssh due to shotgun being run in mcollective 
> >container
>
>
> AFAIK '/var/log/docker-logs/' is available from mcollective container and
> mounted to /var/log/:
>
> [root@fuel-lab-cz5557 ~]# dockerctl shell mcollective mount -l | grep
> os-varlog
> /dev/mapper/os-varlog on /var/log type ext4
> (rw,relatime,stripe=128,data=ordered)
>
> From my experience '/var/log/docker-logs/remote' folder is most ' heavy'
> thing in snapshot.
>
> [0] http://www.gnu.org/software/tar/manual/html_node/dereference.html
>
> Thanks!
>
>
> On 14.01.16 13:00, Igor Kalnitsky wrote:
>
> I took a glance on Maciej's patch and it adds a switch to tar command
> to make it follow symbolic links
>
> Yeah, that should work. Except one thing - we previously had fqdn ->
> ipaddr links in snapshots. So now they will be resolved into full
> copy?
>
>
> I meant that symlinks also give us the benefit of not using additional
> space (just as hardlinks do) while being able to link to files from
> different filesystems.
>
> I'm sorry, I got you wrong. :)
>
> - Igor
>
> On Thu, Jan 14, 2016 at 12:34 PM, Maciej Kwiek <mkw...@mirantis.com> 
> <mkw...@mirantis.com> wrote:
>
> Igor,
>
> I meant that symlinks also give us the benefit of not using additional space
> (just as hardlinks do) while being able to link to files from different
> filesystems.
>
> Also, as Barłomiej pointed out the `h` switch for tar should do the trick
> [1].
>
> Cheers,
> Maciej
>
> [1] http://www.gnu.org/software/tar/manual/html_node/dereference.html
>
> On Thu, Jan 14, 2016 at 11:22 AM, Bartlomiej 
> Piotrowski<bpiotrow...@mirantis.com> <bpiotrow...@mirantis.com> wrote:
>
> Igor,
>
> I took a glance on Maciej's patch and it adds a switch to tar command to
> make it follow symbolic links, so it looks good to me.
>
> Bartłomiej
>
> On Thu, Jan 14, 2016 at 10:39 AM, Igor Kalnitsky <ikalnit...@mirantis.com> 
> <ikalnit...@mirantis.com>
> wrote:
>
> Hey Maceij -
>
>
> About hardlinks - wouldn't it be better to use symlinks?
> This way we don't occupy more space than necessary
>
> AFAIK, hardlinks won't occupy much space. They are the links, after all.
> :)
>
> As for symlinks, I'm afraid shotgun (and fabric underneath) won't
> resolve them and links are get to snapshot As Is. That means if there
> will be no content in the snapshot they are pointing to, they are
> simply useless. Needs to be checked, though.
>
> - Igor
>
> On Thu, Jan 14, 2016 at 10:31 AM, Maciej Kwiek <mkw...@mirantis.com> 
> <mkw...@mirantis.com>
> wrote:
>
> Thanks for your insight guys!
>
> I agree with Oleg, I will see what I can do to make this work this way.
>
> About hardlinks - wouldn't it be better to use symlinks? This way we
> don't
> occupy more space than necessary, and we can link to files and
> directories
> that are in other block device than /var. Please see [1] review for a
> proposed change that introduces symlinks.
>
> This doesn't really give us much right now, because most of the logs
> are
> fetched from master node via ssh due to shotgun being run in
> mcollective
> container, but it's something! When we remove containers, this will
> prove
> more useful.
>

[openstack-dev] [Fuel] Diagnostic snapshot generation is broken due to lack of disk space

[Maciej Kwiek]

Hi!

I need some advice on how to tackle this issue. There is a bug [1]
describing the problem with creating a diagnostic snapshot. The issue is
that /var/log has 100GB available, while /var (where diagnostic snapshot is
being generated - /var/www/nailgun/dump/fuel-snapshot according to [2]) has
10GB available, so dumping the logs can be an issue when logs size exceed
free space in /var.

There are several things we could do, but I am unsure on which course to
take. Should we
a) Allocate more disk space for /var/www (or for whole /var)?
b) Make the snapshot location share the diskspace of /var/log?
c) Something else? What?

Please share your thoughts on this.

Cheers,
Maciej Kwiek

[1] https://bugs.launchpad.net/fuel/+bug/1529182
[2]
https://github.com/openstack/fuel-web/blob/2855a9ba925c146b4802ab3cd2185f1dce2d8a6a/nailgun/nailgun/settings.yaml#L717
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Dropping Python 2.6

[Maciej Kwiek]

+1

On Mon, Dec 14, 2015 at 3:05 PM, Roman Prykhodchenko  wrote:

> Fuelers,
>
> Since Mitaka OpenStack Infra has no resources to test python 2.6 support
> so the corresponding jobs are not running anymore. Since Fuel master node
> is on CentOS 7 now, let’s drop Python 2.6 support in Fuel.
>
>
> - romcheg
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] Patch size limit

[Maciej Kwiek]

Hi,

I recently noticed the influx of big patches hitting Gerrit (especially in
fuel-web, but I also heard that there was a couple of big ones in library).
I think that patches that have 1000 LOC are simply too big to review
thoroughly and reliably.

I would argue that there should be a limit to patch size, either a soft one
(i.e. written down in contributor guidelines) or a hard one (e.g. enforced
by gate job).

I think that we need a discussion whether we really need this limit, what
should it be, and how to enforce it.

I personally think that most patches that are over 400 LOC could be easily
split into at least two smaller changes.

Regarding the limit enforcement - I think we should go with the soft limit,
with X LOC written as a guideline and giving the reviewers a possibility to
give -1s to patches that are too big, but also giving the possibility to
merge bigger changes if it's absolutely necessary (in really rare cases the
changes simply cannot be split). We may mix in the hard limit for
ridiculously large patches (twice the "soft limit" would be good in my
opinion), so the gate would automatically reject such patches, forcing
contributor to split his patch.

Please share your thoughts on this.

Regards,
Maciej Kwiek
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Changing APIs and API versioning

[Maciej Kwiek]

Vitaly,

That's great news. I agree that we need to run fuelclient against every
change in nailgun, but on the other hand - I think we should stick to the
protocol that Igor proposed:

* Announce this change in openstack-dev ML.
* Wait 1 week before approving it, so anyone can prepare.
* Change author has to work with QA in order make sure they are
prepared for this change.

Because some people we may not be aware of can rely on our API behaving in
the way it usually behaves :).

Regards,
Maciej

On Mon, Nov 23, 2015 at 5:46 PM, Vitaly Kramskikh 
wrote:

> Roman,
>
> Sorry for breaking all the stuff again. I've got suggestion to change this
> from one of the core reviewers.
>
> Fortunately, separation of Fuel UI is already in progress (Vladimir
> Kozhukalov may want to provide extra info here), but it won't guarantee
> protection from similar issues in future. What we really need is to run
> fuelclient functional tests against every change in nailgun.
>
> 2015-11-23 22:56 GMT+07:00 Roman Prykhodchenko :
>
>> Folks.
>>
>> This happened again. Nailgun’s API was silently changed [1] breaking
>> python-fuelclient and everything else that was relying on it.
>>
>> I feel like this is the point when just discussing the issue is not
>> enough so I call for a vote: Let’s separate Nailgun from Fuel UI and put
>> them into different repositories now.
>>
>> Please cast your votes (either +1 or -1) to this thread. You can also
>> provide your reasoning and more thoughts.
>>
>>
>> - romcheg
>>
>>
>> 1. https://review.openstack.org/#/c/240234/
>>
>> 26 жовт. 2015 р. о 11:11 Sebastian Kalinowski 
>> написав(ла):
>>
>> 2015-10-23 11:36 GMT+02:00 Igor Kalnitsky :
>>
>>> Roman, Vitaly,
>>>
>>> You're both saying right things, and you guys bring a sore topic up
>>> again.
>>>
>>> The thing is that Nailgun's API isn't the best one.. but we're trying
>>> to improve it step-by-step, from release to release. We have so many
>>> things to reconsider and to fix that it'd require a huge effort to
>>> make backward compatible changes and support it. So we decided ignore
>>> backward compatibility for clients for awhile and consider our API as
>>> unstable.
>>>
>>> I agree with Roman that such changes must not be made secretly, and we
>>> should at least announce about them. Moreover, every core must think
>>> about consequences before approving them.
>>>
>>> So I propose to do the following things when backward incompatible
>>> change to API is required:
>>>
>>> * Announce this change in openstack-dev ML.
>>> * Wait 1 week before approving it, so anyone can prepare.
>>> * Change author has to work with QA in order make sure they are
>>> prepared for this change.
>>>
>>> Any thoughts?
>>>
>>
>>
>> +1.
>>
>> Although there is one thing that you didn't mention (but probably
>> everyone know about it)
>> is to solve the issue with fuelclient not beign tested against changes in
>> nailgun.
>> We need not only run it for every change in nailgun (or for only those
>> that touch files under "api"
>> dir) but also cover more endpoints with fuelclient tests against real
>> API, not mocks, to discover
>> such issues earlier.
>>
>>
>>>
>>> Thanks,
>>> Igor
>>>
>>> On Wed, Oct 21, 2015 at 5:24 PM, Vitaly Kramskikh
>>>  wrote:
>>> > JFYI: (re-)start of this discussion was instigated by merge of this
>>> change
>>> > (and here is revert).
>>> >
>>> > And this is actually not the first time when we make backward
>>> incompatible
>>> > changes in our API. AFAIR, the last time it was also about the network
>>> > configuration handler. We decided not to consider our API frozen, make
>>> the
>>> > changes and break backward compatibility. So now is the time to
>>> reconsider
>>> > this.
>>> >
>>> > API backward compatibility is a must for good software, but we also
>>> need to
>>> > understand that introducing API versioning is not that easy and will
>>> > increase efforts needed to make new changes in nailgun.
>>> >
>>> > I'd go this way:
>>> >
>>> > Estimate the priority of introducing API versioning - maybe we have
>>> other
>>> > things we should invest our resources to
>>> > Estimate all the disadvantages this decision might have
>>> > Fix all the issues in the current API (like this one)
>>> > Implement API versioning support (yes, we don't have this mechanism
>>> yet, so
>>> > we can't just "bump API version" like Sergii suggested in another
>>> thread)
>>> > Consider the current API as frozen v1, so backward incompatible
>>> changes (or
>>> > maybe all the changes?) should go to v2
>>> >
>>> >
>>> > 2015-10-21 20:25 GMT+07:00 Roman Prykhodchenko :
>>> >>
>>> >> Hi folks,
>>> >>
>>> >> I’d like to touch the aspect of Fuel development process that seems
>>> to be
>>> >> as wrong as possible. That aspect is how we change the API.
>>> >>
>>> >> The issue is that in Fuel anyone can change API at any point of time
>>> 

Re: [openstack-dev] [Fuel] Important notice about running tests for python-fuelclient

[Maciej Kwiek]

Missing links from this email:
[1]
https://bitbucket.org/hpk42/tox/issues/285/tox-220-breaks-some-toxini-config-files
[2] https://review.openstack.org/#/c/247452/6

On Mon, Nov 23, 2015 at 12:45 PM, Roman Prykhodchenko  wrote:

> Hi fulers!
> I’d like to let you know that because of the bug [1] in tox 2.2.1 we had
> to change [2] tox.ini temporarily to unlock the gate until the author of
> tox is working on the fix for the problem. Those changes in tox.ini revoke
> all freedom of configuring how tests on one’s local environment are run, so
> all environment variables except FUEL_WEB_ROOT and TEST_NAILGUN_DB are
> ignored.
> Please also note, that setting those two variables is now _mandatory_ —
> tests will fail with strange errors, if that is not done. I will keep you
> updated as soon as tox is fixed and the temporary changes to tox.ini are
> reverted.
>
> - romcheg
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Changing APIs and API versioning

[Maciej Kwiek]

Recently we had some cool work done regarding decoupling repositories from
fuel-web. I think it would be really good to have Fuel UI in separate
repository. +1 to the idea.

On Mon, Nov 23, 2015 at 4:56 PM, Roman Prykhodchenko  wrote:

> Folks.
>
> This happened again. Nailgun’s API was silently changed [1] breaking
> python-fuelclient and everything else that was relying on it.
>
> I feel like this is the point when just discussing the issue is not enough
> so I call for a vote: Let’s separate Nailgun from Fuel UI and put them into
> different repositories now.
>
> Please cast your votes (either +1 or -1) to this thread. You can also
> provide your reasoning and more thoughts.
>
>
> - romcheg
>
>
> 1. https://review.openstack.org/#/c/240234/
>
> 26 жовт. 2015 р. о 11:11 Sebastian Kalinowski 
> написав(ла):
>
> 2015-10-23 11:36 GMT+02:00 Igor Kalnitsky :
>
>> Roman, Vitaly,
>>
>> You're both saying right things, and you guys bring a sore topic up again.
>>
>> The thing is that Nailgun's API isn't the best one.. but we're trying
>> to improve it step-by-step, from release to release. We have so many
>> things to reconsider and to fix that it'd require a huge effort to
>> make backward compatible changes and support it. So we decided ignore
>> backward compatibility for clients for awhile and consider our API as
>> unstable.
>>
>> I agree with Roman that such changes must not be made secretly, and we
>> should at least announce about them. Moreover, every core must think
>> about consequences before approving them.
>>
>> So I propose to do the following things when backward incompatible
>> change to API is required:
>>
>> * Announce this change in openstack-dev ML.
>> * Wait 1 week before approving it, so anyone can prepare.
>> * Change author has to work with QA in order make sure they are
>> prepared for this change.
>>
>> Any thoughts?
>>
>
>
> +1.
>
> Although there is one thing that you didn't mention (but probably everyone
> know about it)
> is to solve the issue with fuelclient not beign tested against changes in
> nailgun.
> We need not only run it for every change in nailgun (or for only those
> that touch files under "api"
> dir) but also cover more endpoints with fuelclient tests against real API,
> not mocks, to discover
> such issues earlier.
>
>
>>
>> Thanks,
>> Igor
>>
>> On Wed, Oct 21, 2015 at 5:24 PM, Vitaly Kramskikh
>>  wrote:
>> > JFYI: (re-)start of this discussion was instigated by merge of this
>> change
>> > (and here is revert).
>> >
>> > And this is actually not the first time when we make backward
>> incompatible
>> > changes in our API. AFAIR, the last time it was also about the network
>> > configuration handler. We decided not to consider our API frozen, make
>> the
>> > changes and break backward compatibility. So now is the time to
>> reconsider
>> > this.
>> >
>> > API backward compatibility is a must for good software, but we also
>> need to
>> > understand that introducing API versioning is not that easy and will
>> > increase efforts needed to make new changes in nailgun.
>> >
>> > I'd go this way:
>> >
>> > Estimate the priority of introducing API versioning - maybe we have
>> other
>> > things we should invest our resources to
>> > Estimate all the disadvantages this decision might have
>> > Fix all the issues in the current API (like this one)
>> > Implement API versioning support (yes, we don't have this mechanism
>> yet, so
>> > we can't just "bump API version" like Sergii suggested in another
>> thread)
>> > Consider the current API as frozen v1, so backward incompatible changes
>> (or
>> > maybe all the changes?) should go to v2
>> >
>> >
>> > 2015-10-21 20:25 GMT+07:00 Roman Prykhodchenko :
>> >>
>> >> Hi folks,
>> >>
>> >> I’d like to touch the aspect of Fuel development process that seems to
>> be
>> >> as wrong as possible. That aspect is how we change the API.
>> >>
>> >> The issue is that in Fuel anyone can change API at any point of time
>> >> without even warning the rest of the same component’s team. Relying on
>> this
>> >> kind of API is basically impossible. We constantly have problems when
>> even
>> >> components of Fuel stop working due to unexpected changes in the API.
>> >> Thinking about another software that must be integrated with Fuel is
>> hardly
>> >> possible with the current approach.
>> >>
>> >> As for a grown-up project there is a strong need for Fuel in general
>> and
>> >> for Nailgun in particular to work on a policy for making changes to
>> their
>> >> APIs. Living in OpenStack ecosystem we must at least take a look how
>> it’s
>> >> done in its components and try to do something similar. After working
>> with
>> >> Nova, Keystone, Ironic and other components I would propose to start
>> with
>> >> the following: let’s not make any changes to the API. Instead, let’s
>> create
>> >> a new version of Nailgun’s API that will 

Re: [openstack-dev] [Fuel][CI] CI on commit message

[Maciej Kwiek]

Hi,

I think that checking commit message compliance to commit message
guidelines (for example ending the first line with dot) is part of CI jobs,
and they will vote -1 if message is wrongly structured.

Maybe there should be separate CI job only for checking commit message?

Cheers,
Maciej Kwiek

On Tue, Jul 28, 2015 at 5:35 PM, Sergii Golovatiuk sgolovat...@mirantis.com
 wrote:

 Hi,

 It looks like our CI is not configured to set +1/-1 automatically, when
 only commit message is changed. It would be nice to have +1 automatically
 when I change Commit Message only instead of waiting hours on CI. It would
 save some hardware resources to save some energy and help environment
 protection movement.

 --
 Best regards,
 Sergii Golovatiuk,
 Skype #golserge
 IRC #holser

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel][CI] CI on commit message

[Maciej Kwiek]

@Aleksandr: it seems you are right, after my first broken commit message I
was careful not to mess them up again and I didn't even notice that this
check was turned off. Just curious: why was it turned off?

On Wed, Jul 29, 2015 at 11:24 AM, Igor Kalnitsky ikalnit...@mirantis.com
wrote:

 Guys,

 How do you suppose to know that only commit message was changed? Do
 you want to implement manual comparison between patch sets?!

 Currently Gerrit checks whether patchset was changed or not by
 tracking Git commit SHA1 sum, and, btw, chaning commit message will
 lead to changing commit sha1 sum.

 So it looks pretty tricky to implement, and it may lead to false
 positive results.

 Thanks,
 Igor

 On Wed, Jul 29, 2015 at 12:06 PM, Sergii Golovatiuk
 sgolovat...@mirantis.com wrote:
  I agree with Maciej
 
  1. Simple change in CommitMessage shouldn't spin up CI check
  2. There should be simple task where we check Implements: blueprint or
  Closes-Bug: or Related-Bug: to set -1 automatically
  3. There can be additional checks like Short Summary should be 50 or less
  symbols. Long summary should be wrapped to 80 symbols
 
 
  --
  Best regards,
  Sergii Golovatiuk,
  Skype #golserge
  IRC #holser
 
  On Wed, Jul 29, 2015 at 10:58 AM, Aleksandr Didenko 
 adide...@mirantis.com
  wrote:
 
  Hi,
 
   I think that checking commit message compliance to commit message
   guidelines (for example ending the first line with dot) is part of CI
 jobs,
   and they will vote -1 if message is wrongly structured.
 
 
  Maciej, we don't have such checks at the moment. You can craft any
 commit
  message you want and it will not cause any problems with CI. I think,
  reviewers could do the job on commit message verification and they
 already
  do this :)
 
  Regards,
  Alex
 
 
  On Wed, Jul 29, 2015 at 11:23 AM, Sergey Vasilenko
  svasile...@mirantis.com wrote:
 
  -1 to Maciej
  +1 to Sergii
 
 
 
  /sv
 
 
 
 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 
 __
  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel]Format of notifications about Ubuntu repositories connectivity

[Maciej Kwiek]

Hi,

I am currently implementing fix for
https://bugs.launchpad.net/fuel/+bug/1439686 .

I plan to notify user about nodes which fail to connect to ubuntu
repositories via fuel notifications. My question is as follows: when I get
the list of nodes which failed repo connectivity test - do I add one
notification for each node, or can I add one big notification, which
consists of all names of all nodes that failed?

What is the general UI strategy for decisions like that?

Cheers,
Maciej Kwiek
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Let's stick to OpenStack global requirements

[Maciej Kwiek]

Just one question regarding this bit:

  There are some plans (unfortunately I do not know details, so maybe
 someone from OSCI could tell more) to split those repositories
 Splitting of repositories doesn't help to solve python packages
 conflicts because master node uses a number of openstack components.
 Anyway it should be done for 7.0 milestone in order to separatre
 master-node distro from environment one. (e.g. if we going to move
 master-node to debian)

Correct me if I am wrong, but isn't it what we have containers on master
node for? We could use their separation to have conflicting versions of
python packages for components that are in separate containers (like
nailgun).

Regards,
Maciej Kwiek




 On Thu, Mar 19, 2015 at 12:01 AM, Dmitry Borodaenko
 dborodae...@mirantis.com wrote:
  Roman,
 
  I like this proposal very much, thanks for the idea and for putting
  together a straightforward process.
 
  I assume you meant: If a requirement that previously was only in Fuel
  Requirements is merged to Global Requirements, it should be removed
  from *Fuel* Requirements.
 
  Sebastian,
 
  We have found ways to resolve the conflicts between clvm and docker,
  and between ruby versions 1.8 and 2.1, without introducing a separate
  package repo for Fuel master. I've updated the blueprint to make note
  of that:
 
 https://blueprints.launchpad.net/fuel/+spec/separate-repo-for-master-node
 
 
 
 
  On Wed, Mar 18, 2015 at 9:25 AM, Sebastian Kalinowski
  skalinow...@mirantis.com wrote:
  I assume that you considered a situation when we have a common
 repository
  with RPMs for Fuel master and for nodes.
  There are some plans (unfortunately I do not know details, so maybe
 someone
  from OSCI could tell more) to split those repositories. How this
 workflow
  will work with those separated repos? Will we still need it?
 
  Thanks!
  Sebastian
 
  2015-03-18 11:04 GMT+01:00 Roman Prykhodchenko m...@romcheg.me:
 
  Hi folks,
 
  before you say «romcheg, go away and never come back again!», please
 read
  the story that caused me to propose this and the proposed solution.
 Perhaps
  it makes you reconsider :)
 
  As you know for different reasons, among which are being able to set up
  everything online and bringing up-to-date packages, we maintain an OSCI
  repository which is used for building ISOs and deploying OpenStack
 services.
  Managing that repo is a pretty hard job. Thus a dedicated group of
 people is
  devoted to perform that duty, they are always busy because of a lot of
  responsibilities they have.
 
  At the same time Fuel’s developers are pretty energetic and always
 want to
  add new features to Fuel. For that they love to use different
 libraries,
  many of which aren’t in the OSCI mirror yet. So they ask OSCI guys to
 add
  more and more of those and I guess that’s pretty fine except one little
  thing — sometimes those libraries conflict with ones, required by
 OpenStack
  services.
 
  To prevent that from happening someone has to check every patch against
  the OSCI repo and OpenStack’s global requirements, to detect whether a
  version bump or adding a new library is required an whether it can be
  performed. As you can guess, there’s too much of a human factor so
  statistically no one does that until problems appear. Moreover, theres’
  nothing but a «it’s not compatible with OpenStack» yelling from OSCI
 team
  that stops developers to change dependencies in Fuel.
 
  All the stuff described above causes sometimes tremendous time losses
 and
  is very problem-prone.
 
  I’d like to propose to make everyone’s life easier by following these
  steps:
 
   - Create a new project called Fuel Requirements, all changes to it
 should
  go through a standard review procedure
   - We strict ourselves to use only packages from both Fuel Requirements
  and Global Requirements for the version of OpenStack, Fuel is
 installing in
  the following manner:
  - If a requirement is in Global Requirements, the version spec in
 all
  Fuel’s components should be exactly like that.
  - OSCI mirror should contain the maximum version of a
 requirement
  that matches its version specification.
  - If a requirement is not in the global requirements list, then
 Fuel
  Requirements list should be used to check whether all Fuel’s components
  require the same version of a library/package.
  - OSCI mirror should contain the maximum version of a
 requirement
  that matches its version specification.
  - If a requirement that previously was only in Fuel Requirements is
  merged to Global Requirements, it should be removed from Global
 Requirements
- Set up CI jobs in both OpenStack CI and FuelCI to check all patches
  against both Global Requirements and Fuel Requirements and block, if
 either
  of checks doesn’t pass
- Set up CI jobs to notify OSCI team if either Global Requirements or
  Fuel Requirements are changed.
- Set up requirements proposal jobs

Re: [openstack-dev] [Fuel] Let's stick to OpenStack global requirements

[Maciej Kwiek]

I guess it would depend on how many docker containers are running on master
node and if we are able to pull off such stunt :).

I am not familiar with the amount of work needed to do sth like that, so
the proposition may be silly. Just let me know if it is.

On Thu, Mar 19, 2015 at 11:51 AM, Dmitry Burmistrov 
dburmist...@mirantis.com wrote:

 Folks,

  Correct me if I am wrong, but isn't it what we have containers on
 master node for?
  On the master node itself conflicts won’t happen because the
 components are run in their containers.

 Do you propose to use _separate_ package repository for each docker
 container? (It means separate gerrit project for each package of each
 container, including openstack projects)

 On Thu, Mar 19, 2015 at 1:16 PM, Roman Prykhodchenko m...@romcheg.me
 wrote:
  Folks,
 
  I assume you meant: If a requirement that previously was only in Fuel
  Requirements is merged to Global Requirements, it should be removed
  from *Fuel* Requirements».
 
  Exactly.
 
  I'm not sure it's good idea.
  We should stay so close to upstream distro as we can. It should be
  very important reason to update package against it's upstream distro
 
 
  The issue is the following: OpenStack’s components are tested against
 those versions of dependencies, that are specified in their requirements.
 IIRC those requirements are set up by pip so CI nodes contain latest
 versions of python packages that match their specs.
 
  The question is whether it’s required to ship OpenStack services with
 packages from a distro or with packages, that are used for testing.
 
  Splitting of repositories doesn't help to solve python packages
  conflicts because master node uses a number of openstack components.
 
  On the master node itself conflicts won’t happen because the components
 are run in their containers.
 
 
  - romcheg
 
 
  19 бер. 2015 о 10:47 Dmitry Burmistrov dburmist...@mirantis.com
 написав(ла):
 
  Roman, all
 
 - OSCI mirror should contain the maximum version of a
 requirement that matches its version specification.
  I'm not sure it's good idea.
  We should stay so close to upstream distro as we can. It should be
  very important reason to update package against it's upstream distro
  version.
  If we update some package, we should maintain it too. Tracking bugs,
  CVEs and so on. The more packages, the more efforts to support it.
 
   - Set up CI jobs to notify OSCI team if either Global Requirements
 or
  Fuel Requirements are changed.
   - Set up requirements proposal jobs that will automatically propose
  changes to all fuel projects once either of requirements lists was
 changed
  Need to be discussed and investigated.
 
  Sebastian, Dmitry,
 
 
  There are some plans (unfortunately I do not know details, so maybe
 someone from OSCI could tell more) to split those repositories
  Splitting of repositories doesn't help to solve python packages
  conflicts because master node uses a number of openstack components.
  Anyway it should be done for 7.0 milestone in order to separatre
  master-node distro from environment one. (e.g. if we going to move
  master-node to debian)
 
 
 
  On Thu, Mar 19, 2015 at 12:01 AM, Dmitry Borodaenko
  dborodae...@mirantis.com wrote:
  Roman,
 
  I like this proposal very much, thanks for the idea and for putting
  together a straightforward process.
 
  I assume you meant: If a requirement that previously was only in Fuel
  Requirements is merged to Global Requirements, it should be removed
  from *Fuel* Requirements.
 
  Sebastian,
 
  We have found ways to resolve the conflicts between clvm and docker,
  and between ruby versions 1.8 and 2.1, without introducing a separate
  package repo for Fuel master. I've updated the blueprint to make note
  of that:
 
 https://blueprints.launchpad.net/fuel/+spec/separate-repo-for-master-node
 
 
 
 
  On Wed, Mar 18, 2015 at 9:25 AM, Sebastian Kalinowski
  skalinow...@mirantis.com wrote:
  I assume that you considered a situation when we have a common
 repository
  with RPMs for Fuel master and for nodes.
  There are some plans (unfortunately I do not know details, so maybe
 someone
  from OSCI could tell more) to split those repositories. How this
 workflow
  will work with those separated repos? Will we still need it?
 
  Thanks!
  Sebastian
 
  2015-03-18 11:04 GMT+01:00 Roman Prykhodchenko m...@romcheg.me:
 
  Hi folks,
 
  before you say «romcheg, go away and never come back again!», please
 read
  the story that caused me to propose this and the proposed solution.
 Perhaps
  it makes you reconsider :)
 
  As you know for different reasons, among which are being able to set
 up
  everything online and bringing up-to-date packages, we maintain an
 OSCI
  repository which is used for building ISOs and deploying OpenStack
 services.
  Managing that repo is a pretty hard job. Thus a dedicated group of
 people is
  devoted to perform that duty, they are always busy because of a lot
 of
  responsibilities they