Woow, long one really. Ok, I think the idea is really good. Of course a lot of work. I as a maintainer of a gnome desktop version for Slackware would like to ask how this would handle the distros which do not use systemd ?
Rgds Saxa www.droplinegnome.org On Tue, Apr 25, 2017 at 4:38 PM, Tristan Van Berkom < tristan.vanber...@codethink.co.uk> wrote: > TL;DR: We are working to improve build tooling for GNOME software > development, and > are very interested in GNOME developer community feedback on the > BuildStream > approach we are taking, and how to achieve a successful transition. > > > Hi all, > > By now many participants of this list are already aware of our efforts > on > the BuildStream tooling from reading my blog posts ([0] and [1]), which we > aspire > to use in GNOME to improve GNOME's developer experience as well as > improving the > process around maintaining GNOME releases. > > At this time I would like to start a more open conversation about our > plans, > ensure that we are all aligned on what the best approach should be for > building > GNOME and also look at how we can implement a better build story for GNOME, > hopefully over the following months. > > There is a lot to say here as it's a big topic, so to structure this a bit, > I'll talk about: > > o What are the existing use cases of building GNOME ? > o What are the pain points related to these use cases ? > o How do we plan to address these pain points using BuildStream ? > o How would we implement BuildStream in GNOME with a smooth transition ? > > > What are the existing use cases of building GNOME ? > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > o A developer uses JHBuild to build and hack on parts of > GNOME on their local laptop or desktop machine, which > can be any linux distro. > > Pain Points: > > * Non deterministic set of dependencies cause builds to fail in > random unpredictable ways, which can be off putting for newcomers > and also an annoyance to seasoned developers who want to fix some > specific bug, but get caught up fixing other bugs instead. > > * It's difficult to debug core components such as the interactions > of gnome-keyring, Linux-PAM, GDM, GNOME Session, GNOME Shell. > > Manual tinkering is needed, you either need a separate machine or > a VM you've setup manually to recreate login scenarios and > gnome-initial-setup scenarios, ensuring a seamless handoff to the > initial user with their online accounts setup and all of this. > > o The release team needs to build and publish GNOME releases > > Pain Points: > > * Non deterministic dependencies again make things unclear as to > what exactly we are releasing. > > E.g., we might know that this vector of GNOME module versions > work well on some specific distro it has been tried with, but > we can't know for sure that the result of a JHBuild of GNOME > will behave the same on any person's distro. > > By the same logic, it becomes difficult as time passes to > build older releases of GNOME in the future on more modern > dependency sets. > > * Current tooling does not allow for any distinction from > a specific version of something (be it a commit sha in a git > repository or a specific tarball), and a symbolic branch name. > > With JHBuild (or flatpak-builder for that matter), you must > either specify a symbolic branch, or a specific commit. > > To advertise a specific release involves a lot of tedious > editing of build metadata to communicate new versions of a stable > or development release set manually. > > o The Flatpak maintainers currently maintain their own set > of build metadata to build the GNOME SDK and runtime. > > Pain Points: > > * Arguably the flatpak maintainership should not be accountable > for maintaining the GNOME Runtime and SDK builds. I think we > mostly agree by now that it would be great to have the GNOME > release team have control of their own flatpak builds. > > There is however a large overlap of libraries and tooling > which must appear in the GNOME Runtimes/SDKs and must also > appear on an operating system running GNOME (libraries and > services to support the display manager, the shell, > the session, control center, etc.). > > Maintaining these sets of build metadata in separate formats > and separate repositories is both burdensome and error prone: > releases are not communicated atomically and nothing guarantees > that GNOME 3.24 moduleset versions and GNOME 3.24 flatpak runtime > versions coincide at all times. > > o Application module maintainers need to either build Flatpaks > of their module or at least provide some flatpak-builder json > for a third party to build it on your behalf. > > Pain Points: > > * Here there is not much in terms of pain points for the author > of a Flatpak application. As long as the only bundled format > your application wants to support is Flatpak, you'll only > need to maintain one set of build metadata. > > o The CI story of building GNOME on automated build machines. > > Pain Points: > > * Here again the problem of maintaining multiple sets of > build metadata in various formats comes up for the release team. > > We should ideally be using the same build metadata for building > GNOME releases as we do for CI. > > > How do we plan to address these pain points using BuildStream ? > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > For Developers > ~~~~~~~~~~~~~~ > First off, BuildStream never allows the build to be contaminated > by host dependencies or tooling in any way. This means that for a given > GNOME release moduleset, every developer will be using exactly the same > set of dependencies and the same tools to build with; any and all > failures > perceived by one developer will be reproducible by any other developer > regardless of what is installed on their host. > > Secondly, in addition to being able to quickly setup a shell environment > and test application code inside a sandbox (similar to how people > generally > use jhbuild), we will also by default provide some build metadata to > allow > users to easily spin up a full VM of GNOME. This means developers can > more > easily test their components against a fully assembled GNOME environment > with the latest master or latest stable of everything in GNOME, also it > will > become a lot easier to hack on core components which deal with the login > scenario (i.e. GNOME Session, GDM, GNOME Keyring etc). > > We care a lot about the developer experience, so our aim is to achieve > a comfortable developer experience for dealing with build failures and > to accelerate your edit/compile/test cycles. We hope that the user will > be able to modify a line of code in glib, and have a new booting VM of > a fully functional GNOME system to test within minutes. > > > For the Release Team > ~~~~~~~~~~~~~~~~~~~~ > > > Deterministic builds > ~~~~~~~~~~~~~~~~~~~~ > There are a few ways that the GNOME Release story can also be improved > by using a virtually entirely deterministic build system. > > o We always know that what is released by GNOME is entirely > reproducible > by anyone who wants to try a release of GNOME. > > o We have much stronger provenance information about not only what > dependencies were chosen to build GNOME on top of, but also how > they were built, when compared to the minimal bound dependency model > we have where JHBuild interrogates the host for a 'sysdep' > > We expect that this will be a great help to any distributions who > wish to integrate GNOME. By examining our own complete reference > builds > it becomes easier to identify integration bugs both on GNOME's side > and > on the given supporting distro's side. > > o All GNOME releases or "reference builds", become entirely repeatable > in the future. This should mean that using build metadata to build > the entire GNOME 3.24.X and create a bootable image should produce > exactly the same thing in 2017 as it will in 2027 > > > Tracking latest branch > ~~~~~~~~~~~~~~~~~~~~~~ > Beyond deterministic builds and repeatability, I should also point out > the `bst track` feature which automates the update of a specific > revision > from a symbolic tracking name (usually a branch). > > So in BuildStream, Source plugins[2] are what import external data in > the > pipeline, usually for the purpose of building, and usually Sources > represent > VCSes or some other means for obtaining external data. Every different > kind > of Source may be attributed both "ref" and "track" attributes: > > o ref: A reference to an exact version, every time the same "ref" is > used, > the imported data should be bit for bit identical as the last > time. > > o track: A symbolic name which the plugin can use to update "ref", > usually > this is a branch name for a VCS. > > For the purpose of running `bst build`, all involved sources must have > a "ref" > already resolved (using git for instance, this means ensuring there > are commit > shas specified for every module you will build). However, as long as > modules > also specify a "track" attribute, their refs can be updated by > invoking `bst track`. > > We expect this can save time for a release team member who might want > to > use the following kind of workflow: > > o Branch the GNOME moduleset repository for a given stable release, > or > use master branch for current dev. > > o Manually specify only the appropriate branch names for each module > in the major point release (e.g.: all GNOME 3.24 branches). > > o Periodically make a release: > > - Checkout the appropriate modulesets branch locally > - run `bst track` on the whole pipeline, automatically updating > the commit shas > - run `bst build` and make sure it at least builds > - git commit the modulesets with the `bst track` results, probably > with > a commit message mentioning the release of GNOME "3.24.4" for > example > > Coincidentally, a lot of the above is also interesting for a CI setup > where > a build machine would: > > o Checkout the appropriate modulesets branch locally > o run `bst track` on the whole pipeline, automatically updating the > commit shas > o run `bst build` and any additional tests on the result > o report any failures > o if the build succeeds, branch, commit and push a release candidate > branch > for human review > > > Build Variants > ~~~~~~~~~~~~~~ > Ideally soon, will come a time where the GNOME release team will be > responsible > for rolling out the GNOME Flatpak SDKs in addition to the regular > release > metadata. > > This basically means for GNOME as a whole, we will start to have > numerous > "build plans" > > Plan A > ~~~~~~ > - Stage a deterministic and bootable base system > (ideally produced by a third party distro) > - Build GNOME platform libraries on top of that > - Build GNOME core modules and services like GDM > - Deploy a bootable disk image > > Plan B > ~~~~~~ > - Stage a deterministic flatpak SDK: freedesktop-sdk-images > - Build some dependencies not provided by freedesktop-sdk-images > - Build GNOME platform libraries on top of that > - Filter and separate parts of the output (the Debug, SDK and > Locale) > - Deploy these separated results to an ostree repo, i.e. a Flatpak > Runtime & SDK > > Our view is that the above two "build plans" can be described in the > same > build metadata (or "BuildStream project"), and that modifying the > upstream > GNOME build metadata in one place should immediately be reflected in > any > builds of either Flatpak runtimes or GNOME bootable images. > > Our vehicle for achieving this is called "variants", the semantics of > "variants" are described in more detail in the documentation[3]. There > are various > ways to achieve configurability in build systems, however we have > found that: > > o In build systems which treat every configuration as a "variable" > and then allow build scripts to make conditional statements on > any variable, we arrive at a combinatorial explosion of complexity > where the project as a whole is virtually impossible to validate > since it purports to support a staggering number of possible > configurations (think of the buildroot configuration system > or gentoo's "use flags"). > > o In build systems which avoid this configurability, we end up > duplicating > the same metadata in cases where one module can be built in more > than > one way, and so we have a lot more metadata to maintain. > > Our concept of "variants" is the compromise, every element may itself > advertise a fixed number of named configurations which it supports and > consequently reduces the number of supported configurations. > Conversely, > one may depend on an element in such a way that is ambivalent of the > variant, which avoids the need for duplication of any build metadata. > > Long story short, we have a very nice plan to allow GNOME to maintain > a tree of build metadata which produces both Flatpak runtimes and > bootable > GNOME systems from the same metadata and software, demonstrating this > is next on our milestones. > > > For CI > ~~~~~~ > Here I am unsure what the best way forward is for CI in GNOME. Clearly > we want to be running installed tests and whatever other tests we can > run on a booted GNOME disk image, and clearly we dont want to maintain > separate sets of build metadata to do any of this. > > We may want to repurpose the existing GNOME continuous framework to > just replace the build procedure with a BuildStream build of the GNOME > release metadata. Alternatively, I think that running a gitlab instance > for the CI cases may get us something superior as we can leverage > a lot of gitlab features and maybe even offer an opt-in pre-merge CI > experience for participating GNOME projects. > > Further than this, I should mention there is some movement to implement > Source plugins to represent distro packaging sources, e.g. the dpkg > source plugin[4]. With this plugin in place, I suspect it will be > very easy to run tests of building and running GNOME against various > third party distributions and versions of those. Imagine we could have > a board somewhere which displays which distributions GNOME builds on > without failing tests (we could always know when GNOME master is failing > against debian sid, or latest fedora, etc). > > > > How would we implement BuildStream in GNOME with a smooth transition ? > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Ideally we would like to demonstrate all of this working in a side branch > of the GNOME release modulesets, without being disruptive or calling for > a flag day where everyone needs to switch over atomically. > > While we were able to achieve something close to that for the Baserock > project which now has some automation to build converted BuildStream > projects beside the YBD tooling, the situation for GNOME is a bit more > complex since we already have two important sources to convert from (the > gnome-sdk-images repository and the release team's JHBuild modulesets). > > So, I'm testing the water in advance to see how the community feels > about this plan in general, and also to gather opinions on what exactly > we should be doing next. > > This is not the whole roadmap[5], but what I have on the menu specifically > for the cases of building GNOME and satisfying GNOME related requirements, > which is my top priority for the next months and listed in the order I > would currently have them done: > > What we have on the menu right now, in the order I would have them done: > > o Complete the Source implementations needed for Flatpak JSON (almost > done, > bzr is supported by flatpak-builder and we need only add some more > tests > to merge bzr support in BuildStream) > > o Complete the Source implementations needed for converting JHBuild > modulesets (I think we will only need svn to complete this list, > possibly > but hopefully not cvs or hg). > > o Experiment with Flatpak deployments > > Most of what BuildStream will have to do to generate Flatpak output for > runtimes and applications is straight forward, just building into some > custom > prefixes and encoding some metadata files for Flatpak to read. > > Looking through flatpak-builder code however there are some > customizations > like post-install manipulation of build output specifically for the > locale > data to end up where Flatpak wants it to be. > > o Create conversion script for JHBuild modulesets. > > Here I would by default use a debian base system to build on top of if > the development of the dpkg plugin[4] works soon. Then I would create > a script which parses the JHBuild XML and outputs a BuildStream > project. > > This is going to need some additional static supporting metadata: > > - We will need to maintain a list of modules (or 'elements') which > should > be built for the GNOME Flatpak SDK/Runtime but not for the GNOME > Releases, > > E.g. non-GNOME dependencies which need to be added to the runtime > but are > not necessarily provided by the freedesktop runtime. > > - Conversely, we need the list of things to be included in GNOME > Releases > but are not built in the GNOME Flatpak runtime/SDK. > > E.g. core GNOME components such as gdm and gnome-session > > With this script in place, we should be able to continuously produce > GNOME > releases which boot in a VM, and also produce the GNOME SDK/runtimes > for Flatpak > without requiring a "flag day" to switch over. > > o Developer Experience > > The developer experience needs work before I can confidently say that > it > can replace JHBuild for the regular day to day requirements of > developers. > > What we absolutely need for this is: > > o Developer Workspaces[6] > > These will allow the developer to override a Source plugin in their > local checkout of the GNOME release metadata, so that for example, > you build everything directly from gits or tarballs, but you build > gedit from a local directory of your choosing. > > o Dual Cache Key Modes[7] > > The shape of this might change a bit, but the gist of it is that > in a release setting, one wants to be sure that everything was > rebuilt > in the case that any of its dependencies have changed (this is > currently > the default for calculating whether something needs to be rebuilt). > > In a developer setting however, you want to go all the way down to > glibc (or lets say glib) and add one printf() statement, or change > a line > of code, and then you want to rebuild only glib and start some > application > or run a fresh new VM with that modified glib, without rebuilding > the > whole world to get there. > > These two scenarios are obviously at odds with eachother so we > will want > some user configuration to make the decision of what to build and > when. > > In addition to the above, it will be nice to also tie up: > > o Artifact Cache Sharing[8] > > Authenticated users or CI machines should be able to upload the > results of individual builds. In this way when a developer wants > to hack on a given module, they really only need to build the > module(s) they are concerned with but can reuse (download) the > build > results of any other builds, provided those were built once and > uploaded to an artifact share. > > o CI Investigation and implementation > > I prefer to keep this as the last piece of the puzzle, as stated above > it's quite possible that we discover more powerful ways to leverage CI > using gitlab, or it may be that refactoring current GNOME Continuous to > build with the same build metadata maintained by the release team is > the way to go. > > In either case I think the requirements here are: > > o Use the same metadata as release team, not a separate format/repo > > o Should be easily repeatable. As in anyone should be able to setup > a new instance to run CI on GNOME in their home by following some > steps in a README somewhere, ideally this should not take all day > and it should be reliably repeatable. > > o Should not have any regressions over the current solution. Which > means installed-tests should run, some screenshots proving a booted > system and some apps, fast build times preferred over rebuilding > everything on every dependency change. > > Asides from the above, it will be nice to: > > o Have also full builds run (i.e. rebuilds of anything with a > changed dependency) > > o Have CI builds contribute to a shared artifact cache which users > can tap into in order to save cycles on their own machines. > > > In Closing > ~~~~~~~~~~ > I am of course very excited about what we can accomplish and I very much > hope that you will share in my enthusiasm. > > Does this look like a lot of work to do ? Yes it does, however there is > already a great deal of work done and momentum is on our side. Thanks to > Codethink sponsoring this project, we also fully expect to have budget to > work on this full time for some months to come, so I'm confident that we > can deliver on our promises. > > Feedback and involvement in any form are greatly appreciated, are > there parts of the picture you think we've missed ? Please reply and > tell us about them :) > > Any questions about what we have created so far and how it works ? Please > reply and ask about these ! > > Best Regards, > -Tristan > > > [0]: https://blogs.gnome.org/tvb/2017/02/06/introducing-buildstream-2/ > [1]: https://blogs.gnome.org/tvb/2017/04/10/buildstream- > progress-and-booting-images/ > [2]: https://buildstream.gitlab.io/buildstream/#sources > [3]: https://buildstream.gitlab.io/buildstream/format.html# > variant-conditionals > [4]: https://gitlab.com/BuildStream/buildstream/issues/10 > [5]: https://wiki.gnome.org/Projects/BuildStream/Roadmap > [6]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/Workspaces > [7]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/CacheKeyModes > [8]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/ArtifactSharing > _______________________________________________ > desktop-devel-list mailing list > desktop-devel-list@gnome.org > https://mail.gnome.org/mailman/listinfo/desktop-devel-list >
_______________________________________________ desktop-devel-list mailing list desktop-devel-list@gnome.org https://mail.gnome.org/mailman/listinfo/desktop-devel-list
