Nice feedback! I've managed to digest a half of it, so let me react to the first half and I will try to follow up with the rest later. :-)
I don't know Solaris IPS (I'll definitely have a look), but from what I understood, it looks like they are delivering system components as multiple repositories. Each repo delivers a component which you can install in "many ways" using something called facets - like devel facet, docs facet, etc. In modularity, we are also (kind of) delivering multiple repositories [1] - even though we are still using RPM packages as a delivery mechanism - (we call these modules) and we can also install them in different way using something called "install profiles". Install profiles define a set of packages from a given module that will be installed on your system. So you could have a "default" install profile, "devel" one for -devel packages, "docs" one for -docs packages, "lang-en_GB" for language, etc. You would be able to install one or more install profiles. We would still (at least for now) use RPM as a delivery mechanism, but from my understanding, it could achieve very similar results. Modules are defined by modulemd [2] - including list of packages, install profiles, etc. What do you think? Cheers! Adam [1] https://asamalik.fedorapeople.org/modularity-building.jpg [2] https://pagure.io/modulemd/blob/master/f/spec.yaml On Thu, Mar 23, 2017 at 9:02 AM, Tomasz Kloczko <kloczko.tom...@gmail.com> wrote: > > OK. Will try to write longer comment :) > Looks like more or less Modularity people are trying to solve problems > already solved in Solaris IPS mediators which allows solve > problems of delivery software in different versions/variants. > Trying to solve such dilemmas by manage as well software which is not > delivered as regular package probably will break completely as > software in working state as non-packaged software will not have proper > requires/provides description. > IPS trying to solve such problems is using know from Linux alternatives > management + dependency resolver. > Highly likely solving multiple versions software delivery issues on top of > raw rpm packages (as they are now) will end up big mess > behind. In other words such support IMO must be anchored way deeper into > PM like it was done in IPS. > Another issue is that some level of flexibility manipulation of versions > of some components may be working only in strictly controlled > sets of versions of the software. > Deliver such such baselines is solved in IPS by incorporations idea which > is mechanism guarantee consistency on some exact areas where > delivery of alternatives is possible. Single incorporation simple locks > all possible to install packages on exact version and releases. > > At the moment similar locking is done in rpm based distros like Fedora is > done by delivery whole distribution. However all internal > dependencies inside exact distro version are based only on > serial:version-release(arch) dependencies and there is no mechanism which > will start delivery alarms or will not allow to install anything from > other versions of the distribution. > There is no in current rpm based approaches mechanisms allowing make > upgrade from distribution version N to version N+1 with > signalisation that some already installed packages are/will be breaking > dependencies encircled on areas on which exact distribution > packages have been tested together, > > Generally delivery packaged software in multiple versions on top of > packages like rpm will be really hard if not impossible, as long > each variant adds yet another dimensions the same stuff which needs to be > delivered. > > This is why IPS completely moved away from packages delivered in form > archives and switched to serving software in form of > repositories. > > IPS mediators + facets ideas really solves Modularity problems and few > other things as well. > Surprisingly something like IPS facet idea in some very limited form is > available on top of rpm. > For example at the moment is possible to choose install everything with or > without documentation (rpm exclude doc mechanism) which is > basing on %doc tokens in %files sections. The same is with choosing > languages/locale dependent files basing on %lang() tokens. > In each of those two "dimensions" are used "dimension" specific %files > tokens. In other words in rpm world is possible to choose within > only those two defined "dimensions". > The same possibility of customisation in case of IPS is delivered in more > general way in form of facet like doc=[true|false], > locale.<lang_name>=[true|false], The same file in package description can > be marked using multiple facets as well. Something ca e > documenttion in exact language. > > How this approach may be used on some other areas? > One example: someone on top of typical system want to compile something > because additionally software must be tsted "in situ". Choosing > some exact set of -devel packages to install to start compiling some > software? No .. just "pkg change-facet devel=true" and within few > minutes ALL already installed pakages will be enreached by adding all > files with devel=true facet. > On Solaris there is no separated devel packages!!!! > > This beautifully as well interacts with AI (Automated installer) manifest > where on specifying installation of the system with exact > attributes is possible nicely described this by: > > <software type="IPS"> > <destination> > <image> > <facet set="false">facet.doc.html</facet> > <facet set="false">facet.doc.pdf</facet> > <facet set="false">facet.locale.*</facet> > <facet set="true">facet.devel</facet> > <facet set="true">facet.locale.en</facet> > <facet set="true">facet.locale.en_GB</facet> > </image> > </destination> > > Solving problems of moving around software in form of archives? No problem > 'software type="ARCHIVE"' (Unified ARchive). For example > initial version of OpenStack packages on Solaris where provided as UAR. > Example from my first experiments with OpenStack trying to use it by > install and setup this software over AI manifest and profile: > > <software type="ARCHIVE"> > <source> > <file uri="http://<hostname>/AI/sol-11_3-openstack-x86.uar"/>; > </source> > <destination> > <image> > <facet set="false">facet.devel</facet> > <facet set="false">facet.doc.html</facet> > <facet set="false">facet.doc.pdf</facet> > <facet set="false">facet.locale.*</facet> > <facet set="true">facet.locale.en</facet> > <facet set="true">facet.locale.en_GB</facet> > </image> > </destination> > > Back to example with compiling something on top of regular system .. > So .. software has been compiled and we have now binaries. > OK. Another command "pkg change-facet devel=false" and all devel stuff is > removed. Someone want to have an access to documentation > during development process? .. easy to guess "pkg change-facet > 'doc*=true'" (there are few doc* facets used in Solaris packages). > It is possible to add dependencies on facet dependent files. > > Few days ago bugs in build-id infrastructure which is now integrated > within each rpm packages kicked hardly. Good that it has been > already solved (partially). > Generally build-id tries to solve delivery of debuginfo packages/resources > for exact and matching versions-releas(arch) for example > core dumping binaries (https://bugzilla.redhat.com/show_bug.cgi?id=1433837 > are some of my comments how build-id could be still > simplified if it would be relaying more on packages database). However > some bugs are still around (https://bugzilla.redhat.com/show_bug > .cgi?id=1434235) > > So how such problem cold be solved using IPS approach? Simple by > debuginfo=true facet on exact set of packages. PM software will find > exact files which needs to be installed on the system. All without > hardcoding as it is now within additional ELF section build-id > hashes. > In repository exact packages will exist with debuginfo=true files however > on exact system image those files may or may not be installed > depends on facet debuginfo=[true|false] state. By addng dependencies > between debuinfo resources in the same way as between regular rpm > packages is possible to 100% reuse dependencies resolver. > > As I've mentioned rpm allows use only two facets-like types of tagging. > Solaris uses more than two hundreds. > Typical dilemma of cutting system image to absolute minimum is solvable by > facets customisation without changing set of installed > packages. No doubts that something like extreme cutting off used disk > space will fail because will not allow to new states of facets > with breaking some dependencies. > > ---- *** ---- > > What I'm trying to tell by above is that technologies like rpm have been > designed on top of quite precise assumptions and used > approaches on solving some scenarios have been architected within scope of > those assumptions. > When now original design design needs to be transformed to handle few new > assumptions, without breaking scaffold of original design > everything likely will end up in form of growing and breaking apart ball > of yeast than something solid. > > Providing software in form of only repositories (and effectively breaking > paradigm of package as file/archive) solves perfectly on > source side providing software for multiple distro versions of within > *single repository*. What it means? No longer updating > repositories addresses on major upgrade -> one less point o fail on whole > upgrade procedure. > The same repository is used as well to provide software for multiple > architectures and packages on repo side are sharing files with the > same checksums. > As long as new version of the package A delivers only one changed file in > new version on new version package everything else on repo > side will be shared between multiple versions of the packages. Packages > can be way bigger and as long new versions of the same packages > will be changeing only some subset of files owned by package over network > automatically will be transferred only what has been changed. > Solutions like drpm (delta rpm) are completely not needed because delta > resources are automatically formed. > > If someone is interested some more details about IPS please try to have > look on source code repo https://java.net/projects/ips/sources/ > pkg-gate/show > IPS code is probably something like +20 times smaller than rpm, dnf and > all additional python modules code combined. Aditionally it > provides all repo side services with caching and providing multi layered > repos infrastructure services software. IPS is fully written > in python and in many places still it solves a lot of more > problems/scenarios which are still ahead of Fedora to solve. > > For example on may systems happens something like this that someone > installs some additional package as JFDI solution. As it was done > during weekend Monday the same person forgets that temporary solution need > to be solved in some clean and tested way. After this > someone else starts using this newly installed software adding to the > system software some script. This scrip have been even added to > install profile used in full OS reinstall/DR recovery procedure. However > after next cycle of reinstallation such script starts failing > and no one remember why and what is needed. > How to avoid such scenarios? Easy: by locking whole set of packages after > initial installation by executing from cron every day > uninstall every package which is not within originally locked set of > packages. By this our example script will fail next day after > installation recalling automatically to finish solving JFDI properly much > earlier. > > Part of the IPS internal simplicity lies as well on top of other OS > provided technologies like using snapshots. Even single new package > installation starts from creating on affecter volumes snapshots. If > package installation fails and it is usually hard to say how to > roll back all changes on PM layer. So .. no problem. Just roll back > everything to checkpointed state in matter of fraction of second. > > The same approach is possible to use on Linux. However to solve this the > same way all ext, xfs and few other FSes needs to be excluded > from new approach and only btrfs ATM could be used as only fully supported > platform. Radical approach .. but 100% it will be working > without breaking internal simplicity. > > What is more important is that on top of IPS have been already proven that > this new approach is working. In other words IPS it is stash > of tested in combat ideas (I'm not suggesting to switch from rpm to IPS > because ~99.99% Linux community ATM is not mentally ready to > start thinking about more radical approaches to PM :) ) > IMO it would be really good if people involved in Modularity will have > closer look on IPS to avoid reinventing the wheel. > Best would be to switch to IPS but probably this time again it will be not > possible to avoid NIH syndrome :) > > Whole and so huge IPS simplification on code layer was possible only > because about decade ago few people come to conclusion that it is > no longer possible to solve new problems using old paradigms of SySV > packages. rpm still sits very hardly on basic SySV packages ideas > which as the set they've been invented *~30 years ago* (a lot of people > here was born around the same time when those fundaments have > been lied :) ). > > As I'm looking one more time now on Modularity I think that those problems > which this project is trying to solve should be handled > exactly in the same way because if not .. whole project IMO has high > chance to fail. > IMO it is only matter of time when rpm will be abandoned because no longer > would be possible to stretch this software onto new needs > without breaking internal consistency. > rpm still is very strong but already with each day is slightly weaker and > weaker. > > kloczek > -- > _______________________________________________ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > -- Adam Šamalík --------------------------- Software Engineer Red Hat
_______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org
