Hi Johann, *>> Where is this requirement coming from? To have multiple storage locations for configurations?* I addition to what is said previously, Another key requirement coming from the concept of "Zero Migration upgrade", which we are trying to achieve as much as possible going forward. Customers already have previous File-Based artifacts should be able to continue as it is, after a product upgrade, as long as their deployment architecture does not change.
Cheers, Ruwan A On Thu, Jul 4, 2019 at 9:03 AM Johann Nallathamby <joh...@wso2.com> wrote: > Ignore the question Isura, I think Ruwan's reply contains the answer. > > Regards, > Johann. > > On Thu, Jul 4, 2019 at 8:48 AM Johann Nallathamby <joh...@wso2.com> wrote: > >> Hi Isura, >> >> On Fri, Jun 7, 2019 at 9:16 AM Isura Karunaratne <is...@wso2.com> wrote: >> >>> >>> >>> On Wed, Jun 5, 2019 at 9:34 AM Ruwan Abeykoon <ruw...@wso2.com> wrote: >>> >>>> Hi All, >>>> The "User Store" configuration can be considered as a deployment >>>> artifact if we look at the following aspects. >>>> >>>> a) "Secondary User Stores" are added, removed and updated per tenant >>>> basis. (Same as SP, and IdP configs) >>>> b) "User Store", "IdP" and "SP", XACML policies, etc behaves as a >>>> collection of business rules, defining the authentication and authorization >>>> flows per the tenant. >>>> c) A change in a particular "User Store" usually affect the >>>> "Authentication" decisions done via SP config. Hence they have tight >>>> coupling. >>>> d) All "User Store", "SP", "IdP", etc, need to be taken as one unit, >>>> when we consider environment to environment promotion of these configs. >>>> (Dev->QA->staging->Prod) >>>> >>>> Hence IMHO, treating "User Store" as the file-based artifact was a >>>> right decision, when our products have been designed for deployment on >>>> bear-metal or VM. However moving forward to container, and cloud >>>> nativeness, posses the challenge on sharing these artifacts. >>>> >>>> Also, considering the CI/CD pipelines, the governance aspect of >>>> changing the configurations, etc, these type of configs need to be >>>> considered as artifacts. We might need to version control these artifacts >>>> in future and may need to push and pull them from VCS systems. >>>> >>>> What we need to do is to have a delegation pattern implemented for all >>>> current file based (and registry based artifacts), where we can switch the >>>> repository from file based one to different system. DB based repository >>>> would be the first such(simple) implementation. We may need to implement >>>> Git based repository when we properly support cloud use cases for example. >>>> We can abstract the storage system, and retain all the parsing and >>>> generation logic unchanged for artifacts. it would be a minimal change and >>>> most versatile way to extend IMO. >>>> >>>> We would need to implement property or "environment variable" binding >>>> logic, to get proper support for environment to environment promotion of >>>> artifacts. yet, it can be done with a separate effort than this IMO. >>>> >>>> Hence +1 to treat >>>> >>>> - Persist data as a blob (marshalled to text form) >>>> - In a separate table structure. >>>> >>>> Cheers, >>>> Ruwan A >>>> >>>> >>>> On Tue, Jun 4, 2019 at 3:50 PM Johann Nallathamby <joh...@wso2.com> >>>> wrote: >>>> >>>>> +1 to get rid of the artifacts for user stores. I think this was a >>>>> wrong decision we made early on. >>>>> >>>>> On Tue, Jun 4, 2019 at 1:19 PM Hasanthi Purnima Dissanayake < >>>>> hasan...@wso2.com> wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> *Problem * >>>>>> Currently, some artifacts like userstores , tenants' data, etc are >>>>>> stored in the file system (not in the database). So when using a >>>>>> clustered >>>>>> setup those artifacts should be shared among all the nodes by using one >>>>>> of >>>>>> the following file sharing mechanisms. >>>>>> >>>>>> - Dep Sync >>>>>> - rSync >>>>>> - Shared File System >>>>>> >>>>>> >>>>>> *Solution * >>>>>> In order to avoid a shared file system and to reduce the deployment >>>>>> and maintenance overhead, those artifacts ca be persisted in the database >>>>>> itself. >>>>>> >>>>>> *Approach* >>>>>> After discussing with @Ruwan Abeykoon <ruw...@wso2.com> and @Isura >>>>>> Karunaratne <is...@wso2.com> we have two options to persist above >>>>>> discussed artifact details. >>>>>> >>>>>> - In the configuration store which is already implemented as >>>>>> discussed in [1][2]. >>>>>> - In a separate table structure. >>>>>> >>>>>> If we are to go with option 01, then we need to consider the >>>>>> artifacts as configurations and persist in the existing schema. >>>>>> >>>>> The advantage of using this is we can re-use the existing >>>>>> implementation including the database schema and existing rest APIs and >>>>>> functionalities (pagination, searching, etc) . >>>>>> >>>>> >>>>> >>>> >>> Hi all, >>> >>> >>>> The drawback is the conceptual difference between an artifact and >>>>>> configuration. >>>>>> >>>>> >>>>> I think this is not a problem. In fact I believe we made a wrong >>>>> decision of considering user stores as artifacts. I can't remember exactly >>>>> as to why we decided like that. User stores are not development artifacts; >>>>> they are one time configurations. They don't have metadata, versioning, >>>>> lifecycles or any other properties associated with other artifacts in >>>>> WSO2. >>>>> >>>>> >>>>>> Further if we are to use the configuration store there is no way to >>>>>> include specific input validations for the user store configuration >>>>>> feature. >>>>>> >>>>> >>>>> I am not sure I completely understand this. But please check the SAML2 >>>>> metadata for identity provider feature where I believe we've done some >>>>> validations on the properties. I think they were done through interceptors >>>>> in identity provider service. >>>>> >>>> Yes. If we have an interception layer, we can do the validation in that >>> level. Currently, we don't have an intercepting layer for configuration >>> management, we can treat this as an improvement to the config management >>> feature. >>> >>> >>>>>> If we are to go with the option 02, then the flow will be as follows. >>>>>> >>>>>> *Existing Flow* >>>>>> >>>>>> [image: Untitled Diagram (9).png] >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *Suggested Flow* >>>>>> >>>>>> [image: Untitled Diagram (10).png] >>>>>> >>>>>> >>>>>> >>>>>> - With the suggested approach, as the storage mechanisms, file >>>>>> system and database can be used and any other storage mechanism is >>>>>> pluggable. >>>>>> >>>>>> I don't agree with classifying user stores as artifacts. Therefore I >>>>> guess for me this option doesn't qualify at all :). >>>>> >>>>> However, just to understand this option better, have we come across >>>>> any customer who has asked us to support multiple types of storage >>>>> mechanisms for artifacts and/or configurations? Where is this requirement >>>>> coming from? I've only seen such requirements for user stores. For >>>>> configurations I feel this is just over engineering. May be it is a valid >>>>> requirement for artifacts? Even if we agree that there are valid reasons >>>>> to >>>>> support this then it has to be supported for all configurations and/or >>>>> artifacts. >>>>> >>>> >> *I think you've missed the important question here :). Where is this >> requirement coming from? To have multiple storage locations for >> configurations?* >> >> Regards, >> Johann. >> >>> >>>>>> - There should be a way to identify the repository where the data >>>>>> is loaded from. The repository can be the file system, database or any >>>>>> other storage mechanism. >>>>>> >>>>>> It sounds like this can get too complicated. >>>>> >>>> I think this is just a flag which we can set while loading the user >>> stores. This will help to show where the user store is configured. Ex. file >>> system, database or GIT location. >>> >>>> >>>>>> - In both the read write operations the enduser should have the >>>>>> control to decide the storage mechanism. >>>>>> >>>>>> Hmm.. this sounds more like a requirement to optimize database read >>>>> write performance. Doesn't sound right for artifacts. >>>>> >>>> This way we can support the backward compatibility. It is true that we >>> can support backward compatibility via a configuration, but this is more >>> flexible. >>> >>>> >>>>>> - If the user needs to migrate a userstore from one storage >>>>>> mechanism (file system) to another then they can do it via UI. >>>>>> >>>>>> Again too many options for the user can make the product fragile. >>>>> >>>> Users may require to migrate the existing file-based configuration to >>> JDBC tables. In such cases, this will help to migrate configuration rather >>> than doing them manually. >>> >>>> >>>>> >>>>>> When persisting the data in the database there are two options we can >>>>>> use : >>>>>> >>>>>> - Persist data as a blob >>>>>> >>>>>> If we persist as blob then we lose the granular control over each >>>>> property for validation, transformation, etc. >>>>> >>>>>> >>>>>> - Persists data as key value pair >>>>>> >>>>>> +1 for this. >>>>> >>>>> >>>>>> If we are to go with the option one then we can persist the file as a >>>>>> blob and reuse most of the existing parsing logics. >>>>>> >>>>> >>>>> Given the understanding I think I prefer option 1 with properties. >>>>> >>>>> Thanks & Regards, >>>>> Johann. >>>>> >>>> >>> Cheers, >>> Isura. >>> >>>> >>>>> >>>>>> >>>>>> Highly appreciate your suggestions and feedbacks on the above >>>>>> approach. >>>>>> >>>>>> [1] [Architecture][IAM][JDBC based Configuration Store] Database >>>>>> Schema >>>>>> [2] [Architecture] [IS] JDBC based Configuration Store for WSO2 IS >>>>>> >>>>>> Thanks, >>>>>> Hasanthi >>>>>> >>>>>> -- >>>>>> >>>>>> Hasanthi Dissanayake | Senior Software Engineer | WSO2 Inc. >>>>>> (m) +94718407133 | (w) +94112145345 | Email: hasan...@wso2.com >>>>>> >>>>>> >>>>> >>>>> -- >>>>> *Johann Dilantha Nallathamby* | Associate Director/Solutions >>>>> Architect | WSO2 Inc. >>>>> (m) +94 (77) 7776950 | (w) +94 (11) 2145345 | (e) joh...@wso2.com >>>>> [image: Signature.jpg] >>>>> >>>> >>>> >>>> -- >>>> Ruwan Abeykoon | Director/Architect | WSO2 Inc. >>>> (w) +947435800 | Email: ruw...@wso2.com >>>> >>>> >>> >>> -- >>> >>> *Isura Dilhara Karunaratne* >>> Technical Lead | WSO2 <http://wso2.com/> >>> *lean.enterprise.middleware* >>> Email: is...@wso2.com >>> Mob : +94 772 254 810 >>> Blog : https://medium.com/@isurakarunaratne >>> >>> >>> >>> >> >> -- >> *Johann Dilantha Nallathamby* | Associate Director/Solutions Architect | >> WSO2 Inc. >> (m) +94 (77) 7776950 | (w) +94 (11) 2145345 | (e) joh...@wso2.com >> [image: Signature.jpg] >> > > > -- > *Johann Dilantha Nallathamby* | Associate Director/Solutions Architect | > WSO2 Inc. > (m) +94 (77) 7776950 | (w) +94 (11) 2145345 | (e) joh...@wso2.com > [image: Signature.jpg] > -- Ruwan Abeykoon | Director/Architect | WSO2 Inc. (w) +947435800 | Email: ruw...@wso2.com
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
