2015-05-27 23:26 GMT+02:00 Derek Higgins <der...@redhat.com>: > On 27/05/15 09:14, Thomas Goirand wrote: >> >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> Hi all, >> >> tl;dr: >> - - We'd like to push distribution packaging of OpenStack on upstream >> gerrit with reviews. >> - - The intention is to better share the workload, and improve the overall >> QA for packaging *and* upstream. >> - - The goal is *not* to publish packages upstream >> - - There's an ongoing discussion about using stackforge or openstack. >> This isn't, IMO, that important, what's important is to get started. >> - - There's an ongoing discussion about using a distribution specific >> namespace, my own opinion here is that using /openstack-pkg-{deb,rpm} or >> /stackforge-pkg-{deb,rpm} would be the most convenient because of a >> number of technical reasons like the amount of Git repository. >> - - Finally, let's not discuss for too long and let's do it!!! :) >> >> Longer version: >> >> Before I start: some stuff below is just my own opinion, others are just >> given facts. I'm sure the reader is smart enough to guess which is what, >> and we welcome anyone involved in the project to voice an opinion if >> he/she differs. >> >> During the Vancouver summit, operation, Canonical, Fedora and Debian >> people gathered and collectively expressed the will to maintain >> packaging artifacts within upstream OpenStack Gerrit infrastructure. We >> haven't decided all the details of the implementation, but spent the >> Friday morning together with members of the infra team (hi Paul!) trying >> to figure out what and how. >> >> A number of topics have been raised, which needs to be shared. >> >> First, we've been told that such a topic deserved a message to the dev >> list, in order to let groups who were not present at the summit. Yes, >> there was a consensus among distributions that this should happen, but >> still, it's always nice to let everyone know. >> >> So here it is. Suse people (and other distributions), you're welcome to >> join the effort. >> >> - - Why doing this >> ================ >> It's been clear to both Canonical/Ubuntu teams, and Debian (ie: myself) >> that we'd be a way more effective if we worked better together, on a >> collaborative fashion using a review process like on upstream Gerrit. >> But also, we'd like to welcome anyone, and especially the operation >> folks, to contribute and give feedback. Using Gerrit is the obvious way >> to give everyone a say on what we're implementing. >> >> As OpenStack is welcoming every day more and more projects, it's making >> even more sense to spread the workload. >> >> This is becoming easier for Ubuntu guys as Launchpad now understand not >> only BZR, but also Git. >> >> We'd start by merging all of our packages that aren't core packages >> (like all the non-OpenStack maintained dependencies, then the Oslo libs, >> then the clients). Then we'll see how we can try merging core packages. >> >> Another reason is that we believe working with the infra of OpenStack >> upstream will improve the overall quality of the packages. We want to be >> able to run a set of tests at build time, which we already do on each >> distribution, but now we want this on every proposed patch. Later on, >> when we have everything implemented and working, we may explore doing a >> package based CI on every upstream patch (though, we're far from doing >> this, so let's not discuss this right now please, this is a very long >> term goal only, and we will have a huge improvement already *before* >> this is implemented). >> >> - - What it will *not* be >> ======================= >> We do not have the intention (yet?) to publish the resulting packages >> built on upstream infra. Yes, we will share the same Git repositories, >> and yes, the infra will need to keep a copy of all builds (for example, >> because core packages will need oslo.db to build and run unit tests). >> But we will still upload on each distributions on separate repositories. >> So published packages by the infra isn't currently discussed. We could >> get to this topic once everything is implemented, which may be nice >> (because we'd have packages following trunk), though please, refrain to >> engage in this topic right now: having the implementation done is more >> important for the moment. Let's try to stay on tracks and be constructive. >> >> - - Let's keep efficiency in mind >> =============================== >> Over the last few years, I've been able to maintain all of OpenStack in >> Debian with little to no external contribution. Let's hope that the >> Gerrit workflow will not slow down too much the packaging work, even if >> there's an unavoidable overhead. Hopefully, we can implement some >> liberal ACL policies for the core reviewers so that the Gerrit workflow >> don't slow down anyone too much. For example we may be able to create >> new repositories very fast, and it may be possible to self-approve some >> of the most trivial patches (for things like typo in a package >> description, adding new debconf translations, and such obvious fixes, we >> shouldn't waste our time). >> >> There's a middle ground between the current system (ie: only write >> access ACLs for git.debian.org with no other check what so ever) and a >> too restrictive fully protected gerrit workflow that may slow down >> everyone too much. Currently, there's a small amount of people involved >> in the packaging. While we can expect a lot of operators will be >> interested to work on core packages like Nova, Neutron and so on, I am >> at the same time expecting that nobody will care about a given indirect >> python module dependency (and I may still continue to be the only one >> working on these...). We really don't want to be stuck in situations >> with nobody reviewing these. >> >> - - /stackforge or /openstack namespace >> ===================================== >> There's been this question floating around. Should we try joining >> directly as part of OpenStack, as many suggested, or should we go >> through a first round (during one dev cycle, until Liberty is released?) >> through stackforge. >> >> I have to admit that I'm not strongly opinionated about this myself. >> Sure, being an official OpenStack big-tent project would be nice, and it >> would feel worm, but also there's the issue that we don't have any >> commits within the project yet, and therefore finding who's core and >> who's the PTL could be an issue if we want to follow the official >> guidelines. >> >> However, without even discussing the topic between us, it's obvious that >> people like James Page (plus others from his team) and myself would be >> core approvers for the Debian packaging. Fedora guys can decide for >> themselves (I must write names I have in mind for the RPM based-OS side >> of things: M. Runge, Haikel Gemmar and Alan Pevec, for example. Please >> add the missing ones here...). >> >> Another thing is that going for stackforge now means that one day, we'll >> have to suffer from a migration to the openstack namespace, which may be >> 1/ a lot of work and 2/ disruptive. Avoiding such a migration would be >> good. >> >> All together, the packaging team will happily accept whatever is decided >> by the TC. What counts for us is that a decision is taken quickly, so >> that we can start implementing it and share our repositories. So, dear >> TC, please let us know ASAP after your next meeting. >> >> Anyhow, please feel free to discuss the issue within the governance >> patch which I started: >> https://review.openstack.org/#/c/185187/ >> >> - - Specific packaging namespace >> ============================== >> Since distribution packaging through Git implies running a lot of git >> repository (one per package), it's been discussed that we may want to >> use a specific namespace for distributions. For example, >> /stackforge-packaging or /openstack-packaging. >> >> Let's keep in mind that, for OpenStack packaging by the distributions, >> we're talking about more than 200 packages. 235 so far on the OpenStack >> packaging group in Debian [1], and it's increasing all the time... >> >> Personally, I believe it'd be nice to have a specifc namespace for >> distributions, for multiple reasons. The first one is that we already >> have a lot of projects inside /stackforge or /openstack, and that >> searching through them isn't easy. Now, consider that we'll have >> duplicates for each and every project. Also, we don't have a complete >> match between OpenStack package names. For example, in Debian/Ubuntu, we >> have "python-" as prefix for each and every python libraries (including >> for example all oslo packages, but not only). Plus we'd be forced to use >> a prefix, for example "pkg-nova", to avoid collision. However, it'd be >> nice to keep the same repository names as per the name of the source >> package in the distributions, otherwise we'd have to fix our tooling >> which obviously will become slightly more complex. >> >> - - Distribution specific git repositories or specific branches >> ============================================================= >> There's 2 issues if we keep the same repository names for multiple >> distribution. >> >> First, we'd like to have a different set of core reviewers for a given >> distribution family. This implies that, if we use the same repository >> for RPM and DEB, then we'd have to set ACLs on per-branch basis, which >> may be harder to maintain (compared to giving access to the full set of >> Git repositories for a given distro family). Yes, we could give access >> to both for core reviewers, and trust them to behave. This trust model >> has been proved as successful within the Debian community, but still, >> adding and removing ACLs for new core reviewers would be harder to >> maintain. >> >> Then, as written above, it is desirable to have a matching pair between >> distribution source package names and the name of the underlying git >> repository (so that for building X, you git clone X, not prefix-X). >> However, distributions don't have the same names for packages. For >> example, RPM supports upper case, while dpkg systems don't. Also, Red >> Hat uses a prefix "openstack-" for all core projects (like >> "openstack-nova"), while Debian based distribution don't, and so on... >> >> - - Finally >> ========= >> Finally, well, please don't discuss the color of the bike-shed for too >> long. We don't really care the color as long as we have a shed to share >> the hosting of our packages Git. >> >> Yes, I'm open to discussing all of the issues above, but if and only if >> it doesn't become a blocker for implementing packaging on upstream >> Gerrit. For the /openstack namespace, can we agree to set a deadline in >> 6 days, before the next TC meeting? We really need to get started soon, >> at the beginning of the liberty cycle, so that we can achieve important >> tasks like merging our source packages between Debian and Ubuntu (for >> example) and still be on-time for the release of Liberty. The task will >> be non-trivial, if we want to accommodate all parties, and retain the >> features our users had enjoyed over the past few years. >> >> Last word: thanks to everyone on the infra team (and others) which were >> supportive of the project. All of this is very exciting, and it's really >> great that this is finally happening. > > > This all sounds great thanks for kicking it off > > Ihar gave some good points of how we are doing most of this in RDO, I'll > summarize this a bit here and add a proposal of how I think we can make some > more progress towards your goal > > As part of the RDO project we have been doing pretty much most of what you > described for the last 6 months, because of the number of git repositories > involved we decided to trial it on github[1] using gerrithub[2] for peer > review. Effectively we now have a yum repository representing the collection > of openstack projects per upstream commit for both Fedora and Centos[3][4] > > After some initial teething problems I think we have settled on a process > that appears to be working. > > The tool which we are using to tie the flow together is called delorean[5], > essentially it monitors git repositories and triggers a build when a commit > occurs on either the upstream project or the packaging itself, it shouldn't > be too hard to expand it to build deb's in what ever way is most > appropriate. The main thing I think that is important here is that we all > agree on a common flow. We will also work on adding support so it can build > packages based on ZUUL REFs so we can support CI triggered by upstream > gerrit. > > > I propose we do the following, > > 1. rename all of the packaging repositories in[1] to prefix them with > something like rdorpm- >
couldn't we use branches and just use rpm- prefix ? If OpenSuse people could give their insight, I prefer sharing a common namespace. > 2. Approach infra to explore the possibility of having a new namespace for > this, I think the number of repositories would justify it (depending on what > dependencies are needed it could be well over 100 per distro), if they agree > then we could give infra control of the "openstack-packages" github > organization, or alternatively create a new one. > +1 > 3. for rpm trunk builds we can point our tools at the new location and > continue as we have been. This would have to be changed in rdoinfo[6] and is > currently RDO specific so so we would need some work here if adding support > to delorean for other distros. > +1 > 4. For deb packages you can create new repositories along side the rdorpm-* > repositories > > From a packaging point of view I think the collaboration may stop there but > from the point of view of flow around the packaging we can (and should) > continue to collaborate e.g. > > 5. Add deb support to delorean, I know of at least one person who has > already explored this (steve cc'd), if delorean is too far off the path of > what we want to achieve and there is a better tool then I'm open to change. > +1 > 6. Explore what ci jobs could be run against the packaging > +1 > The trunk repositories have proven very valuable to us over the last few > months, I hope we can make this work in a way that will be valuable to a > wider audience. How would this sound to you? > Yes, thanks to master builds, we were able to significantly increase robustness of RDO. I encourage everyone to follow that path. H. > thanks, > Derek. > > [1] - https://github.com/openstack-packages > [2] - https://review.gerrithub.io/#/q/project:^openstack-packages/.*,n,z > [3] - http://trunk.rdoproject.org/f21/report.html > [4] - http://trunk.rdoproject.org/centos/report.html > [5] - https://github.com/openstack-packages/delorean > [6] - https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml > > > > >> >> Cheers, >> >> Thomas Goirand (zigo) >> >> [1] >> >> https://qa.debian.org/developer.php?login=openstack-de...@lists.alioth.debian.org >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1 >> >> iQIcBAEBCAAGBQJVZXzYAAoJENQWrRWsa0P+M8gP/21TcEfBs/PTxUFRDMSfP97a >> JCVj5JIy6GCFjDC+8mbX9WGWFEbU0hRn2CG/dRJNbcrkjeQJaDosSd3BMmEAsrQO >> ONyu0Qbm2YgclZ9OyUaewE8NPSH8cGRjxS+IdOm5kzhNqX538ohsxD2XNMknx5rU >> cyq3TTDKWL3ot8+SVdUqfvzmZZdESF+8lucneqOiRk9wU/UVt6fdsovuv3QSqM2h >> k7etO4Oev46nQFCVvSmofv2mfGTWn5iWDbcCj7qf/blI5uiL4GkTkzFbkr/+f2De >> 7C4p9rO8WsEaK+Ptg2hayiOm36i6JPnvO6YSOjm6yTeY4LQ3UkULN2/THvftSkYX >> I+oICj7XbJMhaUDbg9mfJhVZ+O5L0v9YHV2v12NIWMbDGb6EzSYsLqmvANwYN5wM >> Y8zihyETZFYsJVmPdqHN6ZPGeuKAroFo0F5rldMJ/++J5exApf6DJyd+JrE7vmyV >> F3sksS+BhmPgmEBbSvcA0ZP8gqEfPWMK5zT1rWBakKy1OHtDaLwnzsCAkaS6kXUJ >> G45Sr965H6pkO10EMpIbjFeCcVIP7UpmHDxq1PQbhUx4aogjVGA1mIWwuD0Cscaj >> hD18bJyZxe0t0MzhkijxWnQ6holJ1C36SvSD/jha6oZ7k9FHIAwo1MklTJcCEo7J >> PBnrpJTi9IkhDmfp21/s >> =5T0x >> -----END PGP SIGNATURE----- >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev