Hi all, In the last conference call we concluded that some services will be tenant aware. For example the Cassandra bundle won't we deployed in every tenant container because this would not scale. So we now have two options: a tenant aware bundle, and a bundle which is not tenant aware and runs in every tenant container. Question which arise: - when to use which option - what are we going to communicate: is the default tenant aware or not - is it explainable that there are two ways of developing - we should prevent that a bundle is developed as tenant unaware and must be transformed to tenant aware because of scaling issues. Or worse: a tenant unaware bundle can be used in a certain application, but not in another application, because of a different deployment - when we are building an application where all tenants are the same (for example BC), why should we develop tenant unaware bundles (besides the programming model) - ...
@Bram: for the cons you state "I got OOM at 500 tenant with 256M memory in poc setup": what causes this OOM? It seems to me as a pretty simple application which only 500 tenants (and sure, only 256M memory). It is the overhead of 500 containers or something else? What would be the memory footprint for BC assuming all BC specific bundles are tenant unaware, assuming BC also needs memory for caching purposes. How well does it scale when I have a small application (eg openID server) with lots of tenants. A more practical question: assume I have a tenant unaware service with a REST interface. I assume the REST is received by the tenant aware HTTP service (which listens to a certain port) in the tenant aware container. How is this call forwarded to the tenant unaware service? I don't think we currently have a mechanism to go from tenant aware to tenant unaware, where you need to know in what container the tenant lives. @All: What are the thoughts about this? Regards, Mark. -----Original Message----- From: amdatu-developers-bounces at amdatu.org [mailto:[email protected]] On Behalf Of Bram de Kruijff Sent: Tuesday, December 14, 2010 7:33 PM To: amdatu-developers at amdatu.org Subject: Re: [Amdatu-developers] Multi-tenancy (and more) design Hi All, just wanna report I committed a poc for the multi (nested) container scenario into my sandbox. The poc (see readme below) shows a tenant aware useradmin (with multitenant storage) at the platform level. The tenantmanager (also a tenant aware) spawns a nested container for each tenant and publishes only(!) the relevant useradmin for that admin into this container along with a global logservice and some bundles that are 'tenant specific'. Pros: * relatively easy as-full-as-you-want isolation a tenants * a nice 'tenant lifecycle' and layer of indirection for manageability * tenant aware code only at platform layer. app developer need not worry * ...? Cons: * heavy(?) I got OOM at 500 tenant with 256M memory in poc setup * portable to other OSGi impl (?) Should be but needs factory. * ...? Obviously this is just a first poc and needs a lot more testing but in the meanwhile I'd like to hear your thoughts, pros and cons. I'd also like to create a poc using framework hooks to so we can properly evaluate the two solutions. Regards, Bram -------------------------------- PoC for nested Tenant Management -------------------------------- basic design: * TenantManager is a tenant aware service that 1) spawns a new osgi container for each tenant registered 2) delegates platform packages to the tenant container 3) proxies (only relevant!) services (log/useradmin) to the tenant 4) monitors container to take action (eg refresh) 5) TODO: publishes tenant services in parent container (eg a REST service?) * This UserAdmin FS storage is now tenant aware that 1) create a storage dir per tenant (based on id) 2) leverages PAX user admin through extender pattern -> useradmin per tenant -> application dev model in tact implementation notes/todos: * set of delegated packages is static (should be extendable) * list proxied services is static (should be extendable) * currently deploys four bundles into each tenant container - Apache Felix Dependency Manager; - Apache Felix Dependency Manager Shell - Apache Felix Shell Service - Apache Felix Remote Shell - Amdatu UserAdmin Commands * currently proxies global logservice into each tenant container * currently proxies relevant useradmin into tenant container * JVM with Xmx=256 went OOM at about 500 tenants 1) Starting this up: * Copy the amdatu-tm folder into devserver layout * Copy startup from this folder to devserver root * startup 2) inside parent container -> tmcreate 245 tmcreate <tenantId> <tenantName> -> tmcreate 245 jan -> tmlist '123' - 11 '245' - jan -> tmdelete 123 Tenant deleted: 123 note: use integers for tenantid as it is used to register a remote shell on that port note: refresh in the parent container will restart all tenants 3) Inside tenant container * telnet 127.0.0.1 245 Felix Remote Shell Console: ============================ -> ps START LEVEL 1 ID State Level Name [ 0] [Active ] [ 0] System Bundle (3.0.6) [ 1] [Resolved ] [ 1] Apache Felix Dependency Manager (3.0.0.SNAPSHOT) [ 2] [Active ] [ 1] Apache Felix Dependency Manager Shell (3.0.0.SNAPSHOT) [ 3] [Active ] [ 1] Apache Felix Shell Service (1.4.2) [ 4] [Active ] [ 1] Apache Felix Remote Shell (1.1.2) [ 5] [Active ] [ 1] Amdatu UserAdmin Commands (0.1.0.SNAPSHOT) -> uacreate 123 User created: 123 -> ualist '123 -> uadelete 123 User deleted: 123 -> On Fri, Dec 10, 2010 at 3:25 PM, Bram de Kruijff <bdekruijff at gmail.com> wrote: > Hi Ivo, > > Good points, thanks. Please feel free to add things on the wiki. I'm on > mobile atm so just two quicks notes: > > I would not call the management server the provisioning server as > provisioning is only part of its responsibilities. On the other hand I do > like the notion of "provisioning a tenant onto a node"! > > The data sharing use case I would aproach as a data (web) service. Either > owned by one or even 3rd party. This removes the data sharing at the > platform level requirement, which I fear would be a swamp of complexity. At > the same time it opens up the data for reuse for any consumer on the web, so > its much more RESTfull. > > Enjoy the weekend! > Regards, > Bram > > On Dec 10, 2010 2:57 PM, "Ivo Ladage-van Doorn" > <Ivo.Ladage-vanDoorn at gxsoftware.com> wrote: >> Hi Bram, >> >> I think this is a good starting point for the multi-tenancy discussion. >> And of course I have some remarks/additions: >> >> Configuration, Management - I would think that the application should >> support configuration per tenant. For example, if one tenant decides to >> reduce the session timeout to 5 minutes, it should not affect the other >> tenants. Some configuration on the other hand might be shared among all >> tenants hosted by the same application and if so, should be read-only to >> them as changing it would affect other tenants. >> Regarding management; I think tenant management should be separated from >> the Amdatu runtime and managed by the provisioning server. The provisioning >> server needs to be tenant aware, as it should decide what tenants run on >> what nodes and if tenants run in isolates OSGi containers and/or VMs >> (dependent on their contract for example; gold contract = isolated container >> and VM). As a consequence a Tenant in the Amdatu runtime is read-only; it is >> managed by the provisioning server (which may, and probably will, run on the >> Amdatu platform). >> >> Sharing information - Some tenants will want to share content. Think of >> big publishers with multiple magazines. Each magazine would be a tenant, but >> they still want to share articles with each other. There could even be >> authorization involved; one tenant adds articles while other tenants can >> only read them and incorporate them into their website, but not change them. >> >> Licensing - Multiple tenants hosted by the same application may have >> different licenses. Dependent on their license, some >> services/functionality/content will or will not be available to them. >> >> Logging - I think logging should be separated, as each tenant would want >> to see only log entries that are relevant to them. More importantly, log >> files may contain information that are absolutely forbidden to be seen by >> others (as we experienced with some customers; we were not even allowed to >> read log files at all, they first has to be censured) >> >> >> Regards, Ivo >> >> -----Original Message----- >> From: amdatu-developers-bounces at amdatu.org >> [mailto:amdatu-developers-bounces at amdatu.org] On Behalf Of Bram de Kruijff >> Sent: vrijdag 10 december 2010 13:22 >> To: amdatu-developers at amdatu.org >> Subject: [Amdatu-developers] Multi-tenancy (and more) design >> >> Hi all, >> >> I just created a mechanisms section under architecture on the wiki and >> have put down an initial analysis on multi-tenancy. IMHO we must >> address these concerns asap and I would appreciate you feedback and >> contributions :) >> >> Regards, >> Bram >> >> ps. http://www.amdatu.org/confluence/display/Amdatu/Principle+Mechanisms >> ps. Working on a proof of concept for embedded containers. In a >> sandbox near you soon. >> _______________________________________________ >> Amdatu-developers mailing list >> Amdatu-developers at amdatu.org >> http://lists.amdatu.org/mailman/listinfo/amdatu-developers >> >> _______________________________________________ >> Amdatu-developers mailing list >> Amdatu-developers at amdatu.org >> http://lists.amdatu.org/mailman/listinfo/amdatu-developers > _______________________________________________ Amdatu-developers mailing list Amdatu-developers at amdatu.org http://lists.amdatu.org/mailman/listinfo/amdatu-developers
