Forgot to mention an important point that Nuwan mentioned. UI driven configuration will be a requirement in the Cloud first architecture. So we will need a combination of DB and the file system.
The DB will store the configs that will be used to render in the UI allowing users to easily manipulate them centrally. But the only purpose of these is to drive the UI. Once a config is saved in the UI it will actually trigger a file level config change on a given container. So containers will be relying on their local file based configs. This is a nice balance of the best of both worlds. On 14 October 2016 at 10:55, Uvindra Dias Jayasinha <uvin...@wso2.com> wrote: > Let me summarize this thread so far. > > The functionality provided by a DB makes it easy to implement reading, > updating and sharing configs, but there have been some legitimate concerns > that have been raised in this thread regarding that. > > As has already been outlined deployment based configs should be part of > the container and so should be file based. We also have limitations with > certain components such as the Gateway which needs to be deployed in the > DMZ and so cannot connect with a DB in that case. > > At the end of the day a file is a persistent storage just like a DB, > albeit much less sophisticated and localized. So maybe we can achieve some > of the advantages I outlined in my initial mail using file configs. The > underlying storage really does not matter(just an implementation detail) as > long as we can achieve the benefits that we need. > > > On 14 October 2016 at 00:11, Lahiru Sandaruwan <lahi...@wso2.com> wrote: > >> Hi All, >> >> If we try to categorize the configurations we have in C4, using the way >> we manage, >> >> 1. *Runtime configurations* >> - Any config you login to an UI and do >> - E.g. API creation/ subscription in APIM, Mediation flow in >> Integration Server, Service providers/ secondary user stores in IS, >> etc.) >> - We are using file system, direct databases, and registry as per >> the requirements for this type >> 2. *One time configurations* >> - Any one time config you configure using the files >> - E.g. Configs we have in carbon.xml, axis2.xml, user-mgt.xml >> - We only use file system for this type >> >> >> With C5 tenant approach, tenants will get to configure a considerable set >> of configs out of #2, but not all. IMO we should use a database for that >> set only, keeping the other one time configurations in the file system from >> #2. >> >> Even hot deployable files are an option for suitable use cases. (Not sure >> if we planning to move away from hot deployable files, with the container >> adaption). A GUI should be provided for managing them in the database or >> hot deployable files. >> >> In other words, we could move those configurations to #1 from #2. >> Wdyt? >> >> Thanks. >> >> >> >> >> >> On Thu, Oct 13, 2016 at 11:39 AM, Lakmal Warusawithana <lak...@wso2.com> >> wrote: >> >>> >>> >>> On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias <nuw...@wso2.com> wrote: >>> >>>> >>>> >>>> On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera <srin...@wso2.com> >>>> wrote: >>>> >>>>> I believe the plan is that for server configs, there is no (admin) UI >>>>> in C5. It can be changed only via config files. End user scenarios such as >>>>> API publisher might have UIs for configs. ( We should agree here or have >>>>> a >>>>> meeting if we want to change that) >>>>> >>>> >>>> On the API Cloud (and other clouds) we will have tenants changing some >>>> of their configs. So for that we will have to provide a UI since we won't >>>> have any other way to make them change files. >>>> >>> >>> @NuwanD, are these admin portal based config? like defining/changing >>> policies etc? >>> >>> >>>> >>>>> What is an example of case where configs will take effect later? >>>>> >>>>> IMO storing configs in the dtabase make the deployment complicated. >>>>> Also it reduce the scalability as all server point to one DB. I think it >>>>> also make docker usecase complex. >>>>> >>>>> >>>>> --Srinath >>>>> >>>>> On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias <nuw...@wso2.com> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta < >>>>>> lakm...@wso2.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On 12 October 2016 at 14:31, Nuwan Dias <nuw...@wso2.com> wrote: >>>>>>> >>>>>>>> There are challenges when moving configs to the DB. We experienced >>>>>>>> it once when we moved the analytics configs to the registry. And then >>>>>>>> we >>>>>>>> moved it back again to the FS because it was too painful to maintain. >>>>>>>> >>>>>>>> 1. The nodes have to keep polling the DB in a fast enough interval. >>>>>>>> This is a unnecessary performance overhead. Because in practise, >>>>>>>> someone >>>>>>>> will only change these configs once. But to support that use case, we >>>>>>>> have >>>>>>>> to keep polling the DB for life. >>>>>>>> >>>>>>>> 2. Gateways don't have access to the DB. So say you're enabling >>>>>>>> analytics (data publishing). You have to propagate that change to the >>>>>>>> Gateway nodes using some mechanism. And with no clustering on C5, this >>>>>>>> is a >>>>>>>> challenge. >>>>>>>> >>>>>>>> If the objective of this is to make the Cloud (tenant) experience >>>>>>>> better, I think we should just restart the tenant's containers with the >>>>>>>> relevant configs in place. >>>>>>>> >>>>>>> >>>>>>> Still we have a problem with regard to how we are going to allow the >>>>>>> tenants to do the configuration changes. Currently we do it through the >>>>>>> registry which will not work for C5. >>>>>>> >>>>>> >>>>>> Yes, so my idea is to provide a UI to do the configs. Those configs >>>>>> we can store anywhere (maybe in a table) just for the sake of rendering >>>>>> that UI. The product code will still read from the config files. When you >>>>>> apply those configs through the press of a button, the container should >>>>>> get >>>>>> rebuilt and restarted with the necessary modification to the config >>>>>> files. >>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Lakmali >>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> NuwanD. >>>>>>>> >>>>>>>> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta < >>>>>>>> lakm...@wso2.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha < >>>>>>>>> uvin...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Sajith, >>>>>>>>>> >>>>>>>>>> Yes even though the boot up time is not an issue in C5 the other >>>>>>>>>> advantages I have outlined are still there to be gained. There is a >>>>>>>>>> huge >>>>>>>>>> effort we have to do on dev ops side to maintain those images you are >>>>>>>>>> talking about because of having everything at file level. >>>>>>>>>> >>>>>>>>>> Some examples from API Manager I can think of are turning >>>>>>>>>> notifications on/off, enable monetization, enable/disable stats, >>>>>>>>>> configure >>>>>>>>>> work flows, Enable/Disable JWT token header. >>>>>>>>>> >>>>>>>>> >>>>>>>>> +1 to move feature related configurations to the database and make >>>>>>>>> them configurable through the UI. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Lakmali >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 12 October 2016 at 12:58, Sajith Kariyawasam <saj...@wso2.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Uvindra, >>>>>>>>>>> >>>>>>>>>>> With cloud deployment in mind, the idea is to boot up the nodes >>>>>>>>>>> in quick time, therefore the docker images are pre-configured with >>>>>>>>>>> all the >>>>>>>>>>> configuration values, which will speed up the node start up. A >>>>>>>>>>> change of >>>>>>>>>>> configuration means a new docker image will be created with the new >>>>>>>>>>> configs, and re-spawn the cluster. >>>>>>>>>>> >>>>>>>>>>> Therefore, IMO a node restart for a config change is not >>>>>>>>>>> relevant, also no need of a periodic config checks. >>>>>>>>>>> >>>>>>>>>>> Btw, can you give me some example configuration you were >>>>>>>>>>> thinking of? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Sajith >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha < >>>>>>>>>>> uvin...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Was wondering about $subject >>>>>>>>>>>> >>>>>>>>>>>> Traditionally we have stored our product configs, be it >>>>>>>>>>>> carbon.xml, api-manager.xml, identity.xml, etc. at file level. Some >>>>>>>>>>>> configs, such as "port offset" are inherently bound to the server >>>>>>>>>>>> startup >>>>>>>>>>>> so it makes sense for them to be at file level, since they come >>>>>>>>>>>> into affect >>>>>>>>>>>> during the startup. But certain runtime configs actually get >>>>>>>>>>>> engaged only >>>>>>>>>>>> when a given feature is used. But having those configs at file >>>>>>>>>>>> level >>>>>>>>>>>> require a restart for the changes to take affect. In C4 API >>>>>>>>>>>> Manager avoided >>>>>>>>>>>> doing restarts for certain config changes, like adding mediation >>>>>>>>>>>> extensions, by storing them in the registry. >>>>>>>>>>>> >>>>>>>>>>>> For C5 a reusable implementation can exist at each node which >>>>>>>>>>>> periodically reads the table(say once a minute) and updates the >>>>>>>>>>>> config >>>>>>>>>>>> values in memory. Products communicate with this config library to >>>>>>>>>>>> get the >>>>>>>>>>>> values of a given config. So eventually they will read the updated >>>>>>>>>>>> value in >>>>>>>>>>>> a short time. If we were to store at least certain configs at DB >>>>>>>>>>>> level >>>>>>>>>>>> there are several advantages. >>>>>>>>>>>> >>>>>>>>>>>> 1. Eliminate need for a restart for changes to take affect. I >>>>>>>>>>>> realize in C5 a restart is relatively cheap so this might not be a >>>>>>>>>>>> big >>>>>>>>>>>> deal, but you still need someone to initiate the restart after the >>>>>>>>>>>> config >>>>>>>>>>>> change. >>>>>>>>>>>> >>>>>>>>>>>> 2. Since the config DB table has a known structure a UI can be >>>>>>>>>>>> easily developed to do CRUD operations for config changes and used >>>>>>>>>>>> by all >>>>>>>>>>>> products. This is a lot more user friendly than asking users to >>>>>>>>>>>> change >>>>>>>>>>>> files. >>>>>>>>>>>> >>>>>>>>>>>> 3. We can provide a REST API to allow config changes to be done >>>>>>>>>>>> on the DB table alternatively. >>>>>>>>>>>> >>>>>>>>>>>> 4. Simplify dev ops by eliminating complicated puppet config >>>>>>>>>>>> templates that need to constantly maintained with new releases. >>>>>>>>>>>> >>>>>>>>>>>> 5. Since configs are in a central DB its easy to manage them >>>>>>>>>>>> since all nodes will read from the same table. >>>>>>>>>>>> >>>>>>>>>>>> 6. Configs can be backed up by simply backing up the table >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Doing this makes sense for certain use cases of API Manger, I'm >>>>>>>>>>>> sure there maybe similar benefits for other products as well. It >>>>>>>>>>>> may not >>>>>>>>>>>> make sense for all configs but at least for some that govern >>>>>>>>>>>> feature >>>>>>>>>>>> functionality its great to have. WDYT? >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Regards, >>>>>>>>>>>> Uvindra >>>>>>>>>>>> >>>>>>>>>>>> Mobile: 777733962 >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Sajith Kariyawasam >>>>>>>>>>> *Associate Tech Lead* >>>>>>>>>>> *WSO2 Inc.; http://wso2.com <http://wso2.com/>* >>>>>>>>>>> *Committer and PMC member, Apache Stratos * >>>>>>>>>>> *AMIE (SL)* >>>>>>>>>>> *Mobile: 0772269575 <0772269575>* >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> Uvindra >>>>>>>>>> >>>>>>>>>> Mobile: 777733962 >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> Architecture@wso2.org >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Lakmali Baminiwatta >>>>>>>>> Associate Technical Lead >>>>>>>>> WSO2, Inc.: http://wso2.com >>>>>>>>> lean.enterprise.middleware >>>>>>>>> mobile: +94 71 2335936 >>>>>>>>> blog : lakmali.com >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> Architecture@wso2.org >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Nuwan Dias >>>>>>>> >>>>>>>> Software Architect - WSO2, Inc. http://wso2.com >>>>>>>> email : nuw...@wso2.com >>>>>>>> Phone : +94 777 775 729 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> Architecture@wso2.org >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Lakmali Baminiwatta >>>>>>> Associate Technical Lead >>>>>>> WSO2, Inc.: http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> mobile: +94 71 2335936 >>>>>>> blog : lakmali.com >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> Architecture@wso2.org >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Nuwan Dias >>>>>> >>>>>> Software Architect - WSO2, Inc. http://wso2.com >>>>>> email : nuw...@wso2.com >>>>>> Phone : +94 777 775 729 >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> ============================ >>>>> Srinath Perera, Ph.D. >>>>> http://people.apache.org/~hemapani/ >>>>> http://srinathsview.blogspot.com/ >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Nuwan Dias >>>> >>>> Software Architect - WSO2, Inc. http://wso2.com >>>> email : nuw...@wso2.com >>>> Phone : +94 777 775 729 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Lakmal Warusawithana >>> Director - Cloud Architecture; WSO2 Inc. >>> Mobile : +94714289692 >>> Blog : http://lakmalsview.blogspot.com/ >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> -- >> >> Lahiru Sandaruwan >> Associate Technical Lead, >> WSO2 Inc., http://wso2.com >> >> lean.enterprise.middleware >> >> m: +94773325954 >> e: lahi...@wso2.com b: http://lahiruwrites.blogspot.com/ >> in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Regards, > Uvindra > > Mobile: 777733962 > -- Regards, Uvindra Mobile: 777733962
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
