Hi, On Tue, May 22, 2012 at 5:02 PM, Anjana Fernando <[email protected]> wrote:
> Hi, > > After having a chat with Dimuthu, she suggested that we keep a flag in > super tenant's registry mentioning if a specific tenant is having any > tasks. So we can check this at the startup and only load the registries > which has any. So there won't be any overhead of loading all the tenant > registries. So I will go ahead with this approach and implement it. And > also, shall I do the tenant partitioning approach I mentioned earlier? .. > I've added the functionality to load the tenant registries if a specific tenant only have any tasks registered, this information is recorded in super tenant's registry. As per a chat with Azeez, there won't be any tenant partitioning logic done for now, since it has to be properly thought through first. I also double checked the implementation of coordination component and, for multi-tenant scenarios, it still uses a single connection to an external ZooKeeper server, so the number of connections will not increase with the number of tenant's, and there will always be a single connection from a single server to ZooKeeper. So for now, creating a single TaskManager object for each tenant doesn't have any considerable impact, even when the tenant count is relatively large. Cheers, Anjana. > > Cheers, > Anjana. > > > On Tue, May 22, 2012 at 3:40 PM, Anjana Fernando <[email protected]> wrote: > >> Hi Dimuthu, >> >> So, you were actually referring to the part where we list the tasks for a >> specific tenant. Not only the part of retrieving the tenant id list. >> TaskManager is there to listen for any new tasks getting created in the >> cluster, so it should be their up and running from the beginning .. also >> TaskManager is the one who checks if a specific tenant has any tasks to >> schedule at first. That is, to check if a specific tenant has tasks, we >> need to list out the tenant ids and we have to contact its registry to find >> if there are any tasks! (the task definitions are saved in the registry) .. >> So this operation is inevitable .. and that is why I proposed a tenant >> partitioning approach in the previous reply. >> >> Cheers, >> Anjana. >> >> >> On Tue, May 22, 2012 at 3:28 PM, Dimuthu Leelarathne >> <[email protected]>wrote: >> >>> Hi Anjana, >>> >>> Please see the stack trace below. The issue here is we are doing >>> something against the framework. When you are initializing all TaskManagers >>> for all tenants at startup it calls the tenant governance registry of the >>> tenant and loads all the realms in a loop. From UM side there is a DB call >>> going at getTenantUserRealm for each call because it is not initialized >>> before. This accounts for 2001 db calls from UM side only if we have 2000 >>> tenants. >>> >>> It could be even more if we account the registry initialization. What I >>> am saying is, can we only initialize the TaskManagers for only the tenants >>> with Tasks or a better design? >>> >>> >>> at java.lang.Class.forName0(Native Method) >>> at java.lang.Class.forName(Class.java:169) >>> at >>> org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:184) >>> at >>> org.wso2.carbon.user.core.common.DefaultRealm.initializeObjects(DefaultRealm.java:147) >>> at >>> org.wso2.carbon.user.core.common.DefaultRealm.init(DefaultRealm.java:113) >>> at >>> org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:252) >>> at >>> org.wso2.carbon.user.core.common.DefaultRealmService.getTenantUserRealm(DefaultRealmService.java:183) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.init(UserRegistry.java:262) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.<init>(UserRegistry.java:222) >>> at >>> org.wso2.carbon.registry.core.session.UserRegistry.<init>(UserRegistry.java:201) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getUserRegistry(EmbeddedRegistryService.java:426) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getSystemRegistry(EmbeddedRegistryService.java:291) >>> at >>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getGovernanceSystemRegistry(EmbeddedRegistryService.java:516) >>> at >>> org.wso2.carbon.ntask.core.TaskUtils.getGovRegistryForTenant(TaskUtils.java:37) >>> at >>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.getTaskTypesForTenant(TaskServiceImpl.java:104) >>> at >>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.initTaskManagersForTenant(TaskServiceImpl.java:85) >>> at >>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.initTaskManagers(TaskServiceImpl.java:77) >>> at >>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.<init>(TaskServiceImpl.java:63) >>> at >>> org.wso2.carbon.ntask.core.internal.TasksDSComponent.activate(TasksDSComponent.java:68) >>> >>> thanks, >>> dimuthu >>> >>> >>> >>> On Tue, May 22, 2012 at 3:20 PM, Anjana Fernando <[email protected]>wrote: >>> >>>> Hi Dimuthu, >>>> >>>> On Tue, May 22, 2012 at 3:12 PM, Dimuthu Leelarathne <[email protected] >>>> > wrote: >>>> >>>>> Hi Anjana, >>>>> >>>>> >>>>> Say there are 2000 tenants. >>>>> >>>>> This is just one call, but after that you load all 2000 user realms in >>>>> a iterator for each tenant. Meaning another 2000 db calls to check for >>>>> custom realms. Now it is 2001 db calls. >>>>> >>>> >>>> I'm sorry, I didn't understand what "custom realms" are. So the code I >>>> use is like the following, >>>> >>>> Tenant[] tenants = >>>> TasksDSComponent.getRealmService().getTenantManager(). >>>> getAllTenants(); >>>> for (Tenant tenant : tenants) { >>>> tids.add(tenant.getId()); >>>> } >>>> >>>> So you mean, when I do "tenant.getId()" also, there is another database >>>> call going out? .. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> >>>>> thanks, >>>>> dimuthu >>>>> >>>>> >>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, May 22, 2012 at 2:59 PM, Anjana Fernando <[email protected]>wrote: >>>>>>> >>>>>>>> Hi Azeez, >>>>>>>> >>>>>>>> On Tue, May 22, 2012 at 2:27 PM, Afkham Azeez <[email protected]>wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, May 22, 2012 at 2:22 PM, Anjana Fernando >>>>>>>>> <[email protected]>wrote: >>>>>>>>> >>>>>>>>>> Hi Dimuthu, >>>>>>>>>> >>>>>>>>>> On Tue, May 22, 2012 at 2:15 PM, Dimuthu Leelarathne < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Currently >>>>>>>>>>> org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl is >>>>>>>>>>> initializing >>>>>>>>>>> task managers for all tenants at startup. This is against our >>>>>>>>>>> tenant lazy >>>>>>>>>>> loading concepts. >>>>>>>>>>> >>>>>>>>>>> Please move tenant ntask initialization to a >>>>>>>>>>> Axis2ConfigurationContextObserver. This will be called when >>>>>>>>>>> TenanConfiguraitonContext is loaded. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> As I remember, this was already brought up by Azeez earlier. The >>>>>>>>>> rationale is, a scheduled task is not a service. It is not something >>>>>>>>>> that >>>>>>>>>> is activated on-demand by a user in a specific tenant. But is >>>>>>>>>> suppose to >>>>>>>>>> work in the scheduled manner continuously. So when a user in a tenant >>>>>>>>>> schedules a task, it should always run, and should not unload when >>>>>>>>>> the >>>>>>>>>> tenant is unloaded. And when starting the server also, all the tasks >>>>>>>>>> should >>>>>>>>>> continue. >>>>>>>>>> >>>>>>>>> >>>>>>>>> But now we have a major problem if we have say 100,001 tenants. We >>>>>>>>> will create this for all tenants unnecessarily. With the next >>>>>>>>> release, we >>>>>>>>> have tenant partitions defined at the LB level, which relies on the >>>>>>>>> worker >>>>>>>>> nodes to lazy load the tenants. There is no tenant partitioning >>>>>>>>> concept at >>>>>>>>> the worker node level. So, even though cluster as-0-100 loads tenants >>>>>>>>> 0-100, the task thing will load all 100,001! We need to find a >>>>>>>>> solution to >>>>>>>>> this issue. >>>>>>>>> >>>>>>>> >>>>>>>> OK, so this has actually turned into a tenant partitioning problem. >>>>>>>> First of all, when you say initializing a TaskManager, it just means, >>>>>>>> creating an object to represent a tasks for a specific tenant and a >>>>>>>> connection to a centrally managed Group object in ZooKeeper. If the >>>>>>>> tasks >>>>>>>> are not scheduled, it will simply take a block of memory and just sit >>>>>>>> there. And it will not load the tenants, I'm simply retrieving the >>>>>>>> total >>>>>>>> tenant id list and only loading the tenant aware task managers, it >>>>>>>> should >>>>>>>> not load other services and all. And also, the actual tasks will not be >>>>>>>> scheduled in a single server, they will be distributed across the whole >>>>>>>> cluster. >>>>>>>> >>>>>>>> So anyway, to support the tenant partitioning problem. I suggest >>>>>>>> the following. Lets say we have several clusters to represent different >>>>>>>> tenant partitions. For example, 3 clusters, to divide the tenants >>>>>>>> equally. >>>>>>>> When starting the server, we can set a Java system property saying, the >>>>>>>> number of clusters used for tenant partitioning and the partitioning >>>>>>>> cluster the current server will belong to. So for example >>>>>>>> -Dtenant.partition.count=3 -Dmy.tenant.partition=2 (suggest suitable >>>>>>>> names). So with these information at hand, when starting the server, >>>>>>>> we can >>>>>>>> start only the tasks suppose to be scheduled in this server (take >>>>>>>> tenant_id >>>>>>>> % tenant.partition.count and match it to my.tenant.partition), and also >>>>>>>> when a new tenant is joining, we can do the same. Hopefully, this type >>>>>>>> of a >>>>>>>> method can be used for other type of similar services too. I can >>>>>>>> implement >>>>>>>> this in ntask now, if it is an acceptable solution. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Anjana. >>>>>>>> >>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Anjana. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> thanks, >>>>>>>>>>> dimuthu >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Dimuthu Leelarathne >>>>>>>>>>> Technical Lead >>>>>>>>>>> >>>>>>>>>>> WSO2, Inc. (http://wso2.com) >>>>>>>>>>> email: [email protected] >>>>>>>>>>> >>>>>>>>>>> Lean . Enterprise . Middleware >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Anjana Fernando* >>>>>>>>>> Associate Technical Lead >>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>> lean . enterprise . middleware >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Afkham Azeez* >>>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>>>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>>>>>> * <http://www.apache.org/>** >>>>>>>>> email: **[email protected]* <[email protected]>* cell: +94 77 3320919 >>>>>>>>> blog: **http://blog.afkham.org* <http://blog.afkham.org>* >>>>>>>>> twitter: >>>>>>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>>>>>>>> * >>>>>>>>> linked-in: **http://lk.linkedin.com/in/afkhamazeez* >>>>>>>>> >>>>>>>>> * >>>>>>>>> * >>>>>>>>> *Lean . Enterprise . Middleware* >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Anjana Fernando* >>>>>>>> Associate Technical Lead >>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>> lean . enterprise . middleware >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Afkham Azeez* >>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>>>> * <http://www.apache.org/>** >>>>>>> email: **[email protected]* <[email protected]>* cell: +94 77 3320919 >>>>>>> blog: **http://blog.afkham.org* <http://blog.afkham.org>* >>>>>>> twitter: >>>>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>>>>>> * >>>>>>> linked-in: **http://lk.linkedin.com/in/afkhamazeez* >>>>>>> * >>>>>>> * >>>>>>> *Lean . Enterprise . Middleware* >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Associate Technical Lead >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Dimuthu Leelarathne >>>>> Technical Lead >>>>> >>>>> WSO2, Inc. (http://wso2.com) >>>>> email: [email protected] >>>>> >>>>> Lean . Enterprise . Middleware >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Associate Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> Dimuthu Leelarathne >>> Technical Lead >>> >>> WSO2, Inc. (http://wso2.com) >>> email: [email protected] >>> >>> Lean . Enterprise . Middleware >>> >>> >> >> >> -- >> *Anjana Fernando* >> Associate Technical Lead >> WSO2 Inc. | http://wso2.com >> lean . enterprise . middleware >> > > > > -- > *Anjana Fernando* > Associate Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- *Anjana Fernando* Associate Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
