Hi Anjana,
Please see the stack trace below. The issue here is we are doing something
against the framework. When you are initializing all TaskManagers for all
tenants at startup it calls the tenant governance registry of the tenant
and loads all the realms in a loop. From UM side there is a DB call going
at getTenantUserRealm for each call because it is not initialized before.
This accounts for 2001 db calls from UM side only if we have 2000 tenants.
It could be even more if we account the registry initialization. What I am
saying is, can we only initialize the TaskManagers for only the tenants
with Tasks or a better design?
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at
org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:184)
at
org.wso2.carbon.user.core.common.DefaultRealm.initializeObjects(DefaultRealm.java:147)
at
org.wso2.carbon.user.core.common.DefaultRealm.init(DefaultRealm.java:113)
at
org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:252)
at
org.wso2.carbon.user.core.common.DefaultRealmService.getTenantUserRealm(DefaultRealmService.java:183)
at
org.wso2.carbon.registry.core.session.UserRegistry.init(UserRegistry.java:262)
at
org.wso2.carbon.registry.core.session.UserRegistry.<init>(UserRegistry.java:222)
at
org.wso2.carbon.registry.core.session.UserRegistry.<init>(UserRegistry.java:201)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getUserRegistry(EmbeddedRegistryService.java:426)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getSystemRegistry(EmbeddedRegistryService.java:291)
at
org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.getGovernanceSystemRegistry(EmbeddedRegistryService.java:516)
at
org.wso2.carbon.ntask.core.TaskUtils.getGovRegistryForTenant(TaskUtils.java:37)
at
org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.getTaskTypesForTenant(TaskServiceImpl.java:104)
at
org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.initTaskManagersForTenant(TaskServiceImpl.java:85)
at
org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.initTaskManagers(TaskServiceImpl.java:77)
at
org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl.<init>(TaskServiceImpl.java:63)
at
org.wso2.carbon.ntask.core.internal.TasksDSComponent.activate(TasksDSComponent.java:68)
thanks,
dimuthu
On Tue, May 22, 2012 at 3:20 PM, Anjana Fernando <anj...@wso2.com> wrote:
> Hi Dimuthu,
>
> On Tue, May 22, 2012 at 3:12 PM, Dimuthu Leelarathne <dimut...@wso2.com>wrote:
>
>> Hi Anjana,
>>
>>
>> Say there are 2000 tenants.
>>
>> This is just one call, but after that you load all 2000 user realms in a
>> iterator for each tenant. Meaning another 2000 db calls to check for custom
>> realms. Now it is 2001 db calls.
>>
>
> I'm sorry, I didn't understand what "custom realms" are. So the code I use
> is like the following,
>
> Tenant[] tenants = TasksDSComponent.getRealmService().getTenantManager().
> getAllTenants();
> for (Tenant tenant : tenants) {
> tids.add(tenant.getId());
> }
>
> So you mean, when I do "tenant.getId()" also, there is another database
> call going out? ..
>
> Cheers,
> Anjana.
>
>
>> thanks,
>> dimuthu
>>
>>
>>
>>> Cheers,
>>> Anjana.
>>>
>>>
>>>>
>>>>
>>>> On Tue, May 22, 2012 at 2:59 PM, Anjana Fernando <anj...@wso2.com>wrote:
>>>>
>>>>> Hi Azeez,
>>>>>
>>>>> On Tue, May 22, 2012 at 2:27 PM, Afkham Azeez <az...@wso2.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, May 22, 2012 at 2:22 PM, Anjana Fernando <anj...@wso2.com>wrote:
>>>>>>
>>>>>>> Hi Dimuthu,
>>>>>>>
>>>>>>> On Tue, May 22, 2012 at 2:15 PM, Dimuthu Leelarathne <
>>>>>>> dimut...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Currently org.wso2.carbon.ntask.core.service.impl.TaskServiceImpl
>>>>>>>> is initializing task managers for all tenants at startup. This is
>>>>>>>> against
>>>>>>>> our tenant lazy loading concepts.
>>>>>>>>
>>>>>>>> Please move tenant ntask initialization to a
>>>>>>>> Axis2ConfigurationContextObserver. This will be called when
>>>>>>>> TenanConfiguraitonContext is loaded.
>>>>>>>>
>>>>>>>
>>>>>>> As I remember, this was already brought up by Azeez earlier. The
>>>>>>> rationale is, a scheduled task is not a service. It is not something
>>>>>>> that
>>>>>>> is activated on-demand by a user in a specific tenant. But is suppose to
>>>>>>> work in the scheduled manner continuously. So when a user in a tenant
>>>>>>> schedules a task, it should always run, and should not unload when the
>>>>>>> tenant is unloaded. And when starting the server also, all the tasks
>>>>>>> should
>>>>>>> continue.
>>>>>>>
>>>>>>
>>>>>> But now we have a major problem if we have say 100,001 tenants. We
>>>>>> will create this for all tenants unnecessarily. With the next release, we
>>>>>> have tenant partitions defined at the LB level, which relies on the
>>>>>> worker
>>>>>> nodes to lazy load the tenants. There is no tenant partitioning concept
>>>>>> at
>>>>>> the worker node level. So, even though cluster as-0-100 loads tenants
>>>>>> 0-100, the task thing will load all 100,001! We need to find a solution
>>>>>> to
>>>>>> this issue.
>>>>>>
>>>>>
>>>>> OK, so this has actually turned into a tenant partitioning problem.
>>>>> First of all, when you say initializing a TaskManager, it just means,
>>>>> creating an object to represent a tasks for a specific tenant and a
>>>>> connection to a centrally managed Group object in ZooKeeper. If the tasks
>>>>> are not scheduled, it will simply take a block of memory and just sit
>>>>> there. And it will not load the tenants, I'm simply retrieving the total
>>>>> tenant id list and only loading the tenant aware task managers, it should
>>>>> not load other services and all. And also, the actual tasks will not be
>>>>> scheduled in a single server, they will be distributed across the whole
>>>>> cluster.
>>>>>
>>>>> So anyway, to support the tenant partitioning problem. I suggest the
>>>>> following. Lets say we have several clusters to represent different tenant
>>>>> partitions. For example, 3 clusters, to divide the tenants equally. When
>>>>> starting the server, we can set a Java system property saying, the number
>>>>> of clusters used for tenant partitioning and the partitioning cluster the
>>>>> current server will belong to. So for example -Dtenant.partition.count=3
>>>>> -Dmy.tenant.partition=2 (suggest suitable names). So with these
>>>>> information
>>>>> at hand, when starting the server, we can start only the tasks suppose to
>>>>> be scheduled in this server (take tenant_id % tenant.partition.count and
>>>>> match it to my.tenant.partition), and also when a new tenant is joining,
>>>>> we
>>>>> can do the same. Hopefully, this type of a method can be used for other
>>>>> type of similar services too. I can implement this in ntask now, if it is
>>>>> an acceptable solution.
>>>>>
>>>>> Cheers,
>>>>> Anjana.
>>>>>
>>>>>
>>>>>>> Cheers,
>>>>>>> Anjana.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> dimuthu
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Dimuthu Leelarathne
>>>>>>>> Technical Lead
>>>>>>>>
>>>>>>>> WSO2, Inc. (http://wso2.com)
>>>>>>>> email: dimut...@wso2.com
>>>>>>>>
>>>>>>>> Lean . Enterprise . Middleware
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Anjana Fernando*
>>>>>>> Associate Technical Lead
>>>>>>> WSO2 Inc. | http://wso2.com
>>>>>>> lean . enterprise . middleware
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list
>>>>>>> Dev@wso2.org
>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Afkham Azeez*
>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>> * <http://www.apache.org/>**
>>>>>> email: **az...@wso2.com* <az...@wso2.com>* cell: +94 77 3320919
>>>>>> blog: **http://blog.afkham.org* <http://blog.afkham.org>*
>>>>>> twitter:
>>>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
>>>>>> *
>>>>>> linked-in: **http://lk.linkedin.com/in/afkhamazeez*
>>>>>>
>>>>>> *
>>>>>> *
>>>>>> *Lean . Enterprise . Middleware*
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Anjana Fernando*
>>>>> Associate Technical Lead
>>>>> WSO2 Inc. | http://wso2.com
>>>>> lean . enterprise . middleware
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Afkham Azeez*
>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>> * <http://www.apache.org/>**
>>>> email: **az...@wso2.com* <az...@wso2.com>* cell: +94 77 3320919
>>>> blog: **http://blog.afkham.org* <http://blog.afkham.org>*
>>>> twitter:
>>>> **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
>>>> *
>>>> linked-in: **http://lk.linkedin.com/in/afkhamazeez*
>>>> *
>>>> *
>>>> *Lean . Enterprise . Middleware*
>>>>
>>>>
>>>
>>>
>>> --
>>> *Anjana Fernando*
>>> Associate Technical Lead
>>> WSO2 Inc. | http://wso2.com
>>> lean . enterprise . middleware
>>>
>>
>>
>>
>> --
>> Dimuthu Leelarathne
>> Technical Lead
>>
>> WSO2, Inc. (http://wso2.com)
>> email: dimut...@wso2.com
>>
>> Lean . Enterprise . Middleware
>>
>>
>
>
> --
> *Anjana Fernando*
> Associate Technical Lead
> WSO2 Inc. | http://wso2.com
> lean . enterprise . middleware
>
--
Dimuthu Leelarathne
Technical Lead
WSO2, Inc. (http://wso2.com)
email: dimut...@wso2.com
Lean . Enterprise . Middleware
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
