Hi Ruwan, Interestingly we both agree on database persistence, however I am trying to find more conviction on treating Identity Admin UI configurations as "artifacts" and not merely "configurations". I would like to take this opportunity to weigh in my opinion on the matter, just to make sure there is enough reasoning behind this change. I believe this is an important distinction because, 1. If we agree with "artifacts" then we can't stop with user stores, but have to eventually transition all Identity Admin UI configurations to this model. Though there is no technical limitations in this approach, it takes a lot of effort, and that effort better be worth it. 2. It changes the complete outlook of the product, and could differentiate us from competitors, in a good way or bad way.
In my understanding I think there are three *independent* questions to answer. 1. File based persistence vs. Database persistence for these configurations 2. Providing file based representation for configurations 3. Definition of artifacts vs. configurations I think on the first point we agree that database persistence is a better fit for IS, if not for anything, just because all Identity Admin UI configurations except user stores are currently in database for IS and we don't see any major problem with it. I think this is what is being discussed in the mail thread [1], and I tend to agree on most of the points @KasunG Gajasinghe <kas...@wso2.com> is mentioning. On the second point, we already have this in IS, although we currently only have it for SPs; we need to cover the rest as well. I guess we can agree on this need as well, as we've had many enterprise customers who have had strong requirements for import/export, automation, product upgrades, environment migration, etc. and asked for some file format to export to. Having it as a human readable text file makes only makes more sense. However, I don't know if we've figured out how to handle secrets in these files. On the third point, to me "artifacts" mean more than file persistence and file representation. If we go by the definition of, anything that is created by the application user at runtime is an artifact, then I will have to agree that SP, IdP, user stores, XACML and everything else is an artifact. However, in a practical sense, I see some differences between, APIs, mediation sequences, BPEL processes, Siddhi scripts, XACML policies, and service providers or identity providers or user stores. 1. APIs, mediation sequences, BPEL processes, Siddhi scripts, and XACML policies have development time aspect, hence involves development time governance. Where as service providers, identity providers and user stores don't really require that. 2. Artifacts generally have lifecycles, versioning, metadata and properties. Are we sure we want to make all Identity Admin UI configurations also fit into that definition? 3. Do we need change management and continuous delivery for Identity Admin UI configurations like service providers, identity providers and user stores? Have we had any customer request for this level development governance for IS? 4. CI/CD can be generally expected as a key requirement for organizations that aspire to have a good integration or API management story. But I don't think it is right to expect it from an IAM prospect. In most cases IAM is the first product they onboard even before EI or API Manager or other enterprise applications. 5. In 90% of the cases Identity Admin UI configurations are going to be done by one or couple of people in an organization, which is not the case with other artifacts mentioned above. And we don't expect changes to service providers, identity providers and user stores, as frequent as other artifacts. Will this not be considered an overhead for something that one or two people change very rarely? 6. Most Identity Admins are not big fans of developer tools. They prefer the old fashioned management UIs. If so does the whole development governance story have a big impact with developer tools? On Wed, Jun 5, 2019 at 9:34 AM Ruwan Abeykoon <ruw...@wso2.com> wrote: > Hi All, > The "User Store" configuration can be considered as a deployment artifact > if we look at the following aspects. > > a) "Secondary User Stores" are added, removed and updated per tenant > basis. (Same as SP, and IdP configs) > I don't think adding, removing or updating per tenant can be a qualification criteria for artifacts. There are some Resident IdP configurations which are updated per tenant and clearly doesn't qualify as artifacts. E.g. SAML2 SSO Entity ID of the tenant. My concern applies to SPs and IdPs as well. > b) "User Store", "IdP" and "SP", XACML policies, etc behaves as a > collection of business rules, defining the authentication and authorization > flows per the tenant. > XACML, adaptive authentication scripts and in future authentication flow designer output can definitely be treated as artifacts, which like you said define authentication/authorization flows However, if you look at user stores, IdPs and SPs , one could think of them as abstract entities in the flow, but their concrete form will be decided based on configuration that we do. This is analogous to Siddhi queries (artifacts) and output event adaptors (configuration). This is the model I ideally would like to have, which is a mix of the two. One of the important tenets about artifacts is that, the artifact must not change from environment to environment, but the configurations can, and therefore have to be externalized. Do SPs, IdPs and user stores qualify in this sense? > c) A change in a particular "User Store" usually affect the > "Authentication" decisions done via SP config. Hence they have tight > coupling. > Hmm.. can you give an example. I won't expect such a thing to exist, and if it does I would think of it as a design problem. SP shouldn't even know what the user store type is or who are the users who can authenticate to it. All it should care about is whether the IdP can identify a valid user and communicate it to the service provider. > d) All "User Store", "SP", "IdP", etc, need to be taken as one unit, when > we consider environment to environment promotion of these configs. > (Dev->QA->staging->Prod) > Agreed. And that can be done with file based representation and import/export APIs. > Hence IMHO, treating "User Store" as the file-based artifact was a right > decision, when our products have been designed for deployment on bear-metal > or VM. However moving forward to container, and cloud nativeness, posses > the challenge on sharing these artifacts. > This is where database persistence will help I think. > > > Also, considering the CI/CD pipelines, the governance aspect of changing > the configurations, etc, these type of configs need to be considered as > artifacts. We might need to version control these artifacts in future and > may need to push and pull them from VCS systems. > Have we had a significant percentage of IAM customers who have asked for this level of governance in the past? I can't remember any from the top of my head. Or else are we consciously shifting our outlook based on what we believe is the next phase of the product? > > What we need to do is to have a delegation pattern implemented for all > current file based (and registry based artifacts), where we can switch the > repository from file based one to different system. DB based repository > would be the first such(simple) implementation. We may need to implement > Git based repository when we properly support cloud use cases for example. > We can abstract the storage system, and retain all the parsing and > generation logic unchanged for artifacts. it would be a minimal change and > most versatile way to extend IMO. > > We would need to implement property or "environment variable" binding > logic, to get proper support for environment to environment promotion of > artifacts. yet, it can be done with a separate effort than this IMO. > How much difference is there between what you are explaining above as environment specific properties and what we have today as SP, IdP or User Store configurations? Do you think we might have mixed up configurations and what qualify as artifacts originally and that is the problem? Do you think we can cleanly separate out the environment specific properties as SP, IdP and user stores and still have configurations along with artifacts? Sorry if I am asking too many questions, but just want to be able to convince myself that we are doing the right thing here once and for all :) Thanks & Regards, Johann. > > Hence +1 to treat > > - Persist data as a blob (marshalled to text form) > - In a separate table structure. > > Cheers, > Ruwan A > > > On Tue, Jun 4, 2019 at 3:50 PM Johann Nallathamby <joh...@wso2.com> wrote: > >> +1 to get rid of the artifacts for user stores. I think this was a wrong >> decision we made early on. >> >> On Tue, Jun 4, 2019 at 1:19 PM Hasanthi Purnima Dissanayake < >> hasan...@wso2.com> wrote: >> >>> Hi All, >>> >>> *Problem * >>> Currently, some artifacts like userstores , tenants' data, etc are >>> stored in the file system (not in the database). So when using a clustered >>> setup those artifacts should be shared among all the nodes by using one of >>> the following file sharing mechanisms. >>> >>> - Dep Sync >>> - rSync >>> - Shared File System >>> >>> >>> *Solution * >>> In order to avoid a shared file system and to reduce the deployment and >>> maintenance overhead, those artifacts ca be persisted in the database >>> itself. >>> >>> *Approach* >>> After discussing with @Ruwan Abeykoon <ruw...@wso2.com> and @Isura >>> Karunaratne <is...@wso2.com> we have two options to persist above >>> discussed artifact details. >>> >>> - In the configuration store which is already implemented as >>> discussed in [1][2]. >>> - In a separate table structure. >>> >>> If we are to go with option 01, then we need to consider the artifacts >>> as configurations and persist in the existing schema. >>> >> The advantage of using this is we can re-use the existing implementation >>> including the database schema and existing rest APIs and functionalities >>> (pagination, searching, etc) . >>> >> >> >>> The drawback is the conceptual difference between an artifact and >>> configuration. >>> >> >> I think this is not a problem. In fact I believe we made a wrong decision >> of considering user stores as artifacts. I can't remember exactly as to why >> we decided like that. User stores are not development artifacts; they are >> one time configurations. They don't have metadata, versioning, lifecycles >> or any other properties associated with other artifacts in WSO2. >> >> >>> Further if we are to use the configuration store there is no way to >>> include specific input validations for the user store configuration feature. >>> >> >> I am not sure I completely understand this. But please check the SAML2 >> metadata for identity provider feature where I believe we've done some >> validations on the properties. I think they were done through interceptors >> in identity provider service. >> >> >>> >>> If we are to go with the option 02, then the flow will be as follows. >>> >>> *Existing Flow* >>> >>> [image: Untitled Diagram (9).png] >>> >>> >>> >>> >>> >>> >>> *Suggested Flow* >>> >>> [image: Untitled Diagram (10).png] >>> >>> >>> >>> - With the suggested approach, as the storage mechanisms, file >>> system and database can be used and any other storage mechanism is >>> pluggable. >>> >>> I don't agree with classifying user stores as artifacts. Therefore I >> guess for me this option doesn't qualify at all :). >> >> However, just to understand this option better, have we come across any >> customer who has asked us to support multiple types of storage mechanisms >> for artifacts and/or configurations? Where is this requirement coming from? >> I've only seen such requirements for user stores. For configurations I feel >> this is just over engineering. May be it is a valid requirement for >> artifacts? Even if we agree that there are valid reasons to support this >> then it has to be supported for all configurations and/or artifacts. >> >>> >>> - There should be a way to identify the repository where the data is >>> loaded from. The repository can be the file system, database or any other >>> storage mechanism. >>> >>> It sounds like this can get too complicated. >> >>> >>> - In both the read write operations the enduser should have the >>> control to decide the storage mechanism. >>> >>> Hmm.. this sounds more like a requirement to optimize database read >> write performance. Doesn't sound right for artifacts. >> >>> >>> - If the user needs to migrate a userstore from one storage >>> mechanism (file system) to another then they can do it via UI. >>> >>> Again too many options for the user can make the product fragile. >> >> >>> When persisting the data in the database there are two options we can >>> use : >>> >>> - Persist data as a blob >>> >>> If we persist as blob then we lose the granular control over each >> property for validation, transformation, etc. >> >>> >>> - Persists data as key value pair >>> >>> +1 for this. >> >> >>> If we are to go with the option one then we can persist the file as a >>> blob and reuse most of the existing parsing logics. >>> >> >> Given the understanding I think I prefer option 1 with properties. >> >> Thanks & Regards, >> Johann. >> >> >>> >>> Highly appreciate your suggestions and feedbacks on the above approach. >>> >>> [1] [Architecture][IAM][JDBC based Configuration Store] Database Schema >>> [2] [Architecture] [IS] JDBC based Configuration Store for WSO2 IS >>> >>> Thanks, >>> Hasanthi >>> >>> -- >>> >>> Hasanthi Dissanayake | Senior Software Engineer | WSO2 Inc. >>> (m) +94718407133 | (w) +94112145345 | Email: hasan...@wso2.com >>> >>> >> >> -- >> *Johann Dilantha Nallathamby* | Associate Director/Solutions Architect | >> WSO2 Inc. >> (m) +94 (77) 7776950 | (w) +94 (11) 2145345 | (e) joh...@wso2.com >> [image: Signature.jpg] >> > > > -- > Ruwan Abeykoon | Director/Architect | WSO2 Inc. > (w) +947435800 | Email: ruw...@wso2.com > > -- *Johann Dilantha Nallathamby* | Associate Director/Solutions Architect | WSO2 Inc. (m) +94 (77) 7776950 | (w) +94 (11) 2145345 | (e) joh...@wso2.com [image: Signature.jpg]
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev