I've been following this conversation somewhat and would like to throw in some background as a plugin writer.
One thing that concerns me in the current plugin model is the number of XML/text files that need to be edited to deploy my plugin. -applicationContext must be edited to add our PluginMangerImpl. -commands.properties file must be edited to included the permissions for the APIs we contributed. -componentContext.xml & nonossComponentContext.xml must be edited to add our Storage Subsystem Provider API. -log4j-cloud.xml must be edited to ensure that our logger is enabled and logging to our necessary default level. I know our situation is a bit different than the current plug in model, but I think it is something we, as a community, are going to begin seeing more of. For a variety of reasons that I won't get in to right now, our plugin will be closed source and kept separate from the ACS source tree. We want our users to be able to simply drop in our jar file to the CS directory or run and installer and have it picked up by the MS upon a restart. It sounds like what you are proposing here would be very beneficial to plugins that are targeting a deployment model like this. Is this something you're looking/hoping/would like to solve, Darren? -Chris -- Chris Suich chris.su...@netapp.com NetApp Software Engineer Data Center Platforms – Cloud Solutions Citrix, Cisco & Red Hat On Sep 18, 2013, at 6:44 PM, Darren Shepherd <darren.s.sheph...@gmail.com> wrote: > I'm not a committer > > > On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang <frank.zh...@citrix.com> wrote: > >> Well. The codes explain more than words. >> It seems the only extra work is adding a property file that specifies >> parent context and current context name, it's not much complex. >> BTW: any reason for working on repo outside ACS? >> >>> -----Original Message----- >>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] >>> Sent: Wednesday, September 18, 2013 2:43 PM >>> To: dev@cloudstack.apache.org >>> Subject: Re: [PROPOSAL] Modularize Spring >>> >>> If you want to see this all working you can just fetch the "no-at-db4" >>> branch at https://github.com/ibuildthecloud/cloudstack.git >>> >>> Plugin composes multiple modules. If modules are siblings they can't >> inject >>> from each other. But a plugin can augment another module if it chooses >> to. >>> The reality is that the core cloudstack is a tangled mess of >> dependencies such >>> that most of the core code can't be modularized as it stands. So there >> exists a >>> context towards the top of the hierarchy called "core" that a lot of jars >>> contribute to it. Here is the full hierarchy right now. I'll probably >> rename a >>> bunch of things, but this gives you an idea. >>> >>> bootstrap >>> system >>> core >>> allocator >>> allocator-server >>> planner >>> api-planner >>> baremetal-planner >>> explicit-dedication >>> host-anti-affinity >>> implicit-dedication-planner >>> server-planner >>> user-concentrated-pod-planner >>> random-allocator >>> api >>> acl-static-role-based >>> rate-limit >>> server-api >>> user-authenticator-ldap >>> user-authenticator-md5 >>> user-authenticator-plaintext >>> user-authenticator-sha256salted >>> backend >>> alert-adapter-server-backend >>> compute >>> alert-adapter-server-compute >>> baremetal-compute >>> fencer-server >>> investigator-server >>> kvm-compute >>> ovm-compute >>> server-compute >>> xenserver-compute >>> network >>> baremetal-network >>> elb >>> midonet >>> nvp >>> ovs >>> server-network >>> ssp >>> storage >>> alert-adapter-server-storage >>> allocator-storage >>> baremetal-storage >>> secondary-storage >>> server-storage >>> storage-image-default >>> storage-image-s3 >>> storage-image-swift >>> storage-volume-default >>> storage-volume-solidfire >>> template-adapter-server >>> discoverer >>> baremetal-discoverer >>> discoverer-server >>> ovm-discoverer >>> xenserver-discoverer >>> >>> >>> >>> If you look at the baremetal hypervisor plugin that is pretty cross >> cutting to >>> most of ACS. So the modules it contributes to are as follows >>> >>> resources/META-INF/cloudstack/baremetal-storage/spring-context.xml >>> resources/META-INF/cloudstack/baremetal-compute/spring-context.xml >>> resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml >>> resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml >>> resources/META-INF/cloudstack/baremetal-planner/spring-context.xml >>> resources/META-INF/cloudstack/baremetal-network/spring-context.xml >>> >>> So it creates child contextes of storage, compute, network, planner, and >>> discoverer to add its extensions where it needs to be. Additionally, >> you'll notice, >>> it adds some beans to the core context from the file resources/META- >>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is because >> it has >>> some manager class that is used by multiple contexts. >>> >>> Frank, I understand the scare that we are going too complex, but do you >> have >>> some other suggestion? I don't like the idea of one gigantic spring >> context. So I >>> feel I am making it as simple as I can while maintaining some order. >> People >>> just need to create one or more spring xml files and a properties files >> that says >>> where to put it in the hierarchy. >>> >>> Additionally, by putting forcing people to put beans in certains modules >> it >>> forces them to think about what is the role of the code. For example, >> today in >>> ACS the *ManagerImpl classes are a huge mess. They implement too many >>> interfaces and the code crosses to many architectural boundaries. Its >> about >>> time we start splitting things up to be more maintainable. >>> >>> If you have some time, please check out what I have on github. >>> >>> Darren >>> >>> >>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang <frank.zh...@citrix.com> >>> wrote: >>> >>>> I am not against boundary, I am just against making things unnecessary >>>> complex to enable boundary. >>>> If we are going this way, I hope we can make it as much as transparent >>>> to developers. That means, as a developer, all a plugin I need to do >>>> is 1) provide my separate spring xml 2) inject beans I want (valid >>>> beans) in my bean and code business logic 3) compile to a jar and put >>>> to some place designated by CloudStack. That's it. >>>> >>>> I raise this topic because I have seen some projects to create >>>> boundary making things horrible complex. And sometimes developers are >>>> hurt by wrong boundaries, as a result, to overcome these limitations >>>> people write lots of ugly code which makes thing even worse. >>>> >>>> However, I am still worry about if we can make things so simpler. >>>> For example, we may have an orchestration context that contains major >>>> beans needed by almost every plugin, this context can be easily set >>>> as parent context for all plugin contexts when bootstrap. However, if >>>> a plugin A needs to access some bean defined in plugin B, given they >>>> are sibling, how plugin framework resolves the dependency ? >>>> >>>>> -----Original Message----- >>>>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] >>>>> Sent: Wednesday, September 18, 2013 11:53 AM >>>>> To: dev@cloudstack.apache.org >>>>> Subject: Re: [PROPOSAL] Modularize Spring >>>>> >>>>> I'm not for OSGi either, but contexts are quite useful and will lead >>>>> to >>>> better >>>>> things. First off, we don't want one gigantic spring XML config >>>>> file >>>> like we have >>>>> today. I think we can all agree on that. So each plugin will have >>>>> to >>>> supply its >>>>> own XML. So the obstacles you mention, will largely be just that >>>>> for >>>> people. >>>>> >>>>> With Spring it is really simple to just inject dependencies and >>>>> cross architectural boundaries. Its currently everywhere in ACS. >>>>> You can't >>>> just say >>>>> we should review code and make sure nobody doesn't do bad things. A >>>> little bit >>>>> of framework to enforce separation is a good thing. But I'm >>>>> guessing >>>> you will >>>>> disagree with me there. >>>>> >>>>> Here are some random points on why contexts are good. Say I want to >>>>> use Spring AOP or Spring TX in my plugin. With your own context you >>>>> can >>>> ensure >>>>> that you won't screw with anybody else code by accidentally having >>>>> you pointcut match their bean. You don't have to worry about bean >>>>> name >>>> conflicts. >>>>> If two config files specify bean X, Spring will gladly just use the >>>>> last >>>> one. I've >>>>> already found multiply defined beans in ACS, but things still just >>>> happen to work. >>>>> Having multiple contexts also defines initialization order. We can >>>> ensure that >>>>> the framework is loaded and ready before child context are loaded >>>>> and >>>> started. >>>>> (we kind of do this today with ComponentLifeCycle, but its a hack in >>>>> my >>>> mind). >>>>> Additionally, when things start you will know, loading context >>>>> "crapping >>>> plug X". >>>>> If spring fails to initialize, the issue it there. Today, if spring >>>> fails to start, it >>>>> could be one of over 500 beans causing the weird problem. The list >>>>> goes >>>> on >>>>> and on. >>>>> >>>>> Finally, this is the big one and why I really want contexts. I have >>>> some notes on >>>>> the wiki [1] that you might want to read through. Basically I want >>>>> to >>>> get to a >>>>> more flexible deployment model that allows both a single monolithic >>>>> JVM >>>> as >>>>> today and also a fully distributed system. Having contexts in >>>>> hierarchy >>>> will >>>>> enable me to accomplish this. By selecting which contexts are >>>>> loaded at runtime will determine what role the JVM takes on. The >>>>> contexts help >>>> people >>>>> better understand how the distributed architecture will work too, >>>>> when >>>> we get >>>>> there. >>>>> >>>>> Frank, trust me, I hate complex things. I don't want OSGi, >>>>> classloader >>>> magic, >>>>> etc. But I do like organization and a little bit of framework so >>>>> that >>>> people don't >>>>> accidentally shoot themselves in the foot. I personally like >>>>> knowing >>>> that I >>>>> couldn't have screwed something up, because the framework won't even >>>> allow >>>>> it. If we separate everything as I want today, and then tomorrow we >>>>> say >>>> this is >>>>> way too much overhead, moving to a flat context is simple. Don't >>>>> think >>>> we are >>>>> on some slippery slope to classloaders and dependency hell. >>>>> >>>>> Darren >>>>> >>>>> [1] >>>>> >>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+ >>>> her >>>>> e...#Nothingtoseehere...-DeploymentModels >>>>> >>>>> >>>>> >>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang >>>>> <frank.zh...@citrix.com>wrote: >>>>> >>>>>> What's the point in using separate spring context per plugin? >>>>>> Separate class loader is the thing I hate most in OSGI, I am >>>>>> afraid we are on the same way. >>>>>> Frankly speaking, I never see benefits of this *separate* model, >>>>>> our project(or most projects) is not like Chrome which has to >>>>>> create sandbox for extensions in order to avoid bad plugin screws >>>>>> up the whole browser(however, I still see bad plugins screw up my >>>>>> Chrome well). >>>>>> Projects like CloudStack using plugin to decouple architecture >>>>>> should not introduce many isolations to plugin writer, the point >>>>>> preventing wrong use of some components is not making much sense >>>>>> to me. If a plugin not following guide(if we have >>>>>> it) we should kick it out, instead of making obstacles for 99% >>>>>> good >>>> people. >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Darren Shepherd [mailto:darren.s.sheph...@gmail.com] >>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM >>>>>>> To: dev@cloudstack.apache.org >>>>>>> Subject: Re: [PROPOSAL] Modularize Spring >>>>>>> >>>>>>> Right, component isn't a thing. I probably shouldn't use that >> term. >>>>>>> I >>>>>> want to >>>>>>> standarize on the naming convention of plugin, module, and >> extension. >>>>>> It is >>>>>>> explained a bit on the wiki [1] but I'll try to do a little >>>>>>> better job >>>>>> here. So a >>>>>>> plugin is basically a jar. A jar contains multiple modules. A >>>>>>> modules >>>>>> ends up >>>>>>> being a spring application context that composes multiple >>>>>>> configuration >>>>>> files. >>>>>>> Modules are assembled into a hierarchy at runtime. Extensions >>>>>>> are implementations of interfaces that exist in a module. A >>>>>>> maven project produces a jar, so a plugin ends up being a maven >>>>>>> project >>>> also. >>>>>>> >>>>>>> So currently today we don't have a strong definition of Plugin >>>>>>> and I >>>>>> hope to >>>>>>> address that. >>>>>>> >>>>>>> Darren >>>>>>> >>>>>>> [1] >>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug- >>>>>>> ins%2C+Modules%2C+and+Extensions >>>>>>> >>>>>>> >>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland >>>>>>> <daan.hoogl...@gmail.com>wrote: >>>>>>> >>>>>>>> sounds great Darren, >>>>>>>> >>>>>>>> By component, you mean maven project or some larger chunk like >>>>>>>> distribution package? (did i miss this definition somewhere or >>>>>>>> do we define the components now?) >>>>>>>> >>>>>>>> regards, >>>>>>>> Daan >>>>>>>> >>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd >>>>>>>> <darren.s.sheph...@gmail.com> wrote: >>>>>>>>> Currently ACS code is fairly modular in that you can add >>>>>>>>> plug-ins to ACS >>>>>>>> to >>>>>>>>> extend most functionality. Unfortunately ACS is not >>>>>>>>> packaged in a >>>>>>>> modular >>>>>>>>> way. It is still delivered essentially as one large unit. >>>>>>>>> There are >>>>>>>> many >>>>>>>>> reason for this but one large barrier to modularizing ACS is >>>>>>>>> that the Spring configuration is managed as one large unit. >>>>>>>>> >>>>>>>>> I propose that we break apart the Spring XML configuration >>>>>>>>> such that each component contributes its own configuration. >>>>>>>>> Additionally each component will be loaded into its own >>>>>>>>> Spring ApplicationContext such that its beans will not >>>>>>>>> conflict with the wiring of other beans in ACS. This change >>>>>>>> will >>>>>>>>> lay the foundation for a richer plugin model and >>>>>>>>> additionally a more distributed architecture. >>>>>>>>> >>>>>>>>> The technical details for this proposal can be found on the >>>>>>>>> wiki >>>> [1]. >>>>>>>>> >>>>>>>>> Darren >>>>>>>>> >>>>>>>>> [1] >>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular >>>>>>>> ize+ >>>>>>>> Spri >>>>>>>> ng >>>>>>>> >>>>>> >>>> >>
