I've finally compiled a spec for this topic https://review.openstack.org/#/c/281557/
On Wed, Feb 17, 2016 at 2:13 PM Alex Schultz <aschu...@mirantis.com> wrote: > On Wed, Feb 17, 2016 at 10:23 AM, Bogdan Dobrelya <bdobre...@mirantis.com> > wrote: > >> > So we'll have tons of conditionals in composition layer, right? Even if >> > some puppet-openstack class have just one new parameter in new release, >> > then we'll have to write a conditional and duplicate class declaration. >> Or >> > write complex parameters hash definitions/merges and use >> > create_resources(). The more releases we want to support the more >> > complicated composition layer will become. That won't make contribution >> to >> > fuel-library easier and even can greatly reduce development speed. Also >> are >> > we going to add new features to stable releases using this workflow with >> > single composition layer? >> >> As I can see from an example composition [0], such code would be an >> unmaintainable burden for development and QA process. Next imagine a >> case for incompatible *providers* like network transformations - shall >> we put multiple if/case to the ruby providers as well?.. >> >> That is not a way to go for a composition, sorry. While the idea may be >> doable, I agree, but perhaps another way. >> > > So I agree that the provided example isn't exactly the cleanest > implementation but I think the point is that we need to start considering > supporting multiple versions of OpenStack for a given set of fuel-library > manifests. I think the provided example begins to show some issues with the > way we currently structure our modular tasks. We basically don't have any > organized structure which leads to having to sprinkle random if/elses > throughout the code to try and provide some semblance of backwards > compatibility or deprecation for tasks. As I see it today in fuel-library > we are too closely tied to the current development version of OpenStack and > with every release we are not providing proper deprecation for > configuration tasks between each release of fuel. I'm sure fuel plugin > developers could probably attest to the amount of pain each release brings > trying to re-reverse engineer the deployment process to see what has > changed. > > I think it's unrealistic to expect that all users are going to want the > latest version of OpenStack as soon as it's released. Fuel is > an OpenStack deployment tool, so why can't we deploy different versions > with a single version of fuel? Technically we should have a way to do with > with releases in Fuel but it seems that we've handicapped any ability to > try and leverage this information to support a different version of > OpenStack. Much like the UCA work where we started to extract out the > package/repo information, we need to extract out the OpenStack release > information (version/repos/etc). We should allow a user to specify that > they want a build a cloud using X fuel release to deploy Y os with Z > OpenStack release. If we can't provide this type of flexibility I'm not > seeing why someone would want to use Fuel over their own OS > provisioning+puppet/ansible/chef deployment method. > > > >> (tl;dr) >> By the way, this reminded me "The wrong abstraction" [1] article and >> discussion. I agree with the author and believe one should not group >> code (here it is versioned puppet modules & compositions) in a way which >> introduces abstractions (here a super-composition) with multiple >> if/else/case and hardcoded things to switch the execution flow based on >> version of things. Just keep code as is - partially duplicated by >> different releases in separate directories with separate modules and >> composition layers and think of better solutions please. >> > > Completely duplicating the fuel-library for each OpenStack release is > probably even more unmanageable then beginning to structure our composition > layer into something that supports multiple OpenStack resources but I guess > that could be an option... > > >> >> There is also a nice comment: "...try to optimize my code around >> reducing state, coupling, complexity and code, in that order". I >> understood that like a set of "golden rules": >> - Make it coupled more tight to decrease (shared) state >> - Make it more complex to decrease coupling >> - Make it duplicated to decrease complexity (e.g. abstractions) >> >> (tl;dr, I mean it) >> So, bringing those here. >> - The shared state is perhaps the Nailgun's world view of all data and >> versioned serializers for supported releases, which know how to convert >> the only latest existing data to any of its supported previous versions. >> - Decoupling we do by putting modules with its compositions to different >> versioned /etc/puppet subdirectories. I'm not sure how do we decouple >> Nailgun serializers though. >> - Complexity is how we compose those modules / write logic of serializers. >> - Duplication is puppet classes (and providers) with slightly different >> call parameters from a version to version. Sometimes even not backwards >> compatible. Probably same to the serializers? >> >> So, we're going to *increase complexity* by introducing >> super-compositions for multi OpenStack releases. Not sure about what to >> happen to the serializers, any volunteers to clarify an impact?. And the >> Rules "allow" us to do so only in order to decrease either coupling or >> shared state, which is not the case, AFAICT. Modules with compositions >> are separated well by OpenStack versions, nothing to decrease. Might >> that change to decrease a shared state? I'm not sure if it even applies >> here. Puppet versioning shares nothing. Only Nailgun folks may know the >> answer. >> >> > I don't think we have to increase complexity if we properly structure > fuel-library. I think we could also structure it in a way that can be > tested in an automated fashion. I think we can come up with a set of rules > for tasks and how to implement a new task that can reduce complexity and > actually allow for code reuse. If you take a look at our tasks today there > is actually an excessive amount of duplication between tasks[0][1] around > variable and configuration gathering before we even call the upstream > OpenStack modules. The tasks we have today are basically just undocumented > freeform puppet that can only be fully tested via multiple deployment tests > because of the lack of decent test coverage. Additionally, we have no real > way of testing deprecation between releases today because there's no formal > api contract for modular tasks themselves. > > I would be interested in investigating a restructure of the tasks that > might go something like.... > 1) Restructure fuel-library tasks into a configuration > generation/formatting method (extracting data from hiera/globals/etc) and > the actual configuration application logic > 2) For the configuration gathering items that we traditionally load up at > the top of our tasks, we could investigate leveraging hiera for providing > data to classes or structure into an osnailyfacter::config::<thing> class. > 2) Move the actual configuration application logic into an > osnailyfacter::task::<thing> method with a proper documented api contract > with deprecation policy. These classes could use something like > create_resources(...) to dynamically load the classes with the provided > parameters to support the class api changes between upstream module > versions. > 3) Add release specific overrides into > osnailyfacter::task::<thing>::<release> where ::<release> could be > automatically included based on something like hiera('openstack_version') > > There are a few added benefits of restructuring the tasks into traditional > puppet classes. > 1) If we need to support any sort of traditional puppet master LCM for > nodes, by moving the tasks into specific class we can actually just > leverage our already written task code to ensure configurations. > 2) Testability around idempotency as we can write beaker tests to be able > to test deployment tasks for idempotency and deployment without having to > stand up all the other pieces of fuel. Similar to the Puppet OpenStack's > puppet-openstack-integration[2] module. > 3) Better defined separation and containment of release specific items so > when we are moving forward. It's much easier to remove n-X release as we > could just 'find deployment/puppet/osnailyfacter/manifests/ -name 'kilo.pp' > -delete' > > That being said, this would be a lot of work but it's mostly taking what > we have today and reorganizing it rather than having to write things from > scratch and providing some policy rules around structure of tasks. I think > in the long run we could benefit by being able to test fuel-library tasks > with existing tools rather than continually having to rely on noop or > actual deployment tests with nailgun/astute/isos/etc. This is just a > starting point for a conversation and I think we should have a serious > discussion about this topic and not just dismiss it because it's hard or > might add complexity. > > > Thanks, > -Alex > > [0] > https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/glance/glance.pp#L3-L99 > [1] > https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/modular/keystone/keystone.pp#L3-L115 > [2] https://github.com/openstack/puppet-openstack-integration/ > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- -- Andrew Woodward Mirantis Fuel Community Ambassador Ceph Community
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev