Hi Gayan, On Mon, Oct 13, 2014 at 7:50 PM, Gayan Gunarathne <gay...@wso2.com> wrote:
> IMO it is based on how many siblings we have in the topology which we can > lock independently. If we have lot of occurrences where we can able to > lock the siblings independently it will improve the performance.But again I > think we need to careful with the dead lock scenarios. > I did not understand your statement exactly. Can you please explain a bit more? My understanding is, if CC allows only sequential access to its own Topology model, then that will again be a bottleneck since CC is the one who sends out the Topology events. > > Thanks, > Gayan > > On Mon, Oct 13, 2014 at 5:50 PM, Isuru Haththotuwa <isu...@apache.org> > wrote: > >> Another improvement that we can do is to introduce the same hierarchical >> locking to the Topology structure maintained in the Cloud Controller. Its >> actually the CC who will be updating the local Topology and sending the >> relevant events. In that case, if we have have only one write lock for the >> whole Topology in the CC's TopologyBuilder, it will still be a bottleneck. >> WDYT? >> >> On Thu, Oct 9, 2014 at 3:36 PM, Isuru Haththotuwa <isu...@apache.org> >> wrote: >> >>> Hi, >>> >>> >>> Since the Topology is updated from the messaging component only, I >>> removed the methods to access the write locks to an internal class. Those >>> methods will not be exposed to outside by the messaging component. Now, the >>> following methods to obtain read-only locks are exposed from >>> TopologyManager: >>> >>> // Topology level read locks >>> >>> /** >>> * Acquires read lock for the Complete Topology >>> */ >>> * public static void acquireReadLock() ;* >>> >>> /** >>> * Releases read lock for the Complete Topology >>> */ >>> * public static void releaseReadLock();* >>> >>> // Application and Service read locks >>> >>> /** >>> * Acquires read lock for the all Applications >>> */ >>> * public static void acquireReadLockForApplications() ;* >>> >>> /** >>> * Releases read lock for the all Applications >>> */ >>> * public static void releaseReadLockForApplications();* >>> /** >>> * Acquires read lock for the all Services >>> */ >>> * public static void acquireReadLockForServices();* >>> >>> /** >>> * Releases read lock for the all Services >>> */ >>> * public static void releaseReadLockForServices() ;* >>> >>> /** >>> * Acquires read lock for a Service >>> * >>> * @param serviceName service name to acquire read lock >>> */ >>> * public static void acquireReadLockForService (String serviceName) ;* >>> >>> /** >>> * Releases read lock for a Service >>> * >>> * @param serviceName service name to release read lock >>> */ >>> * public static void releaseReadLockForService (String serviceName) ;* >>> >>> /** >>> * Acquires read lock for a Cluster. This will acquire the read lock >>> in the following order >>> * 1. for the Service >>> * 2. for the Cluster >>> * >>> * @param serviceName service name to acquire read lock >>> * @param clusterId cluster id to acquire read lock >>> */ >>> * public static void acquireReadLockForCluster (String serviceName, >>> String clusterId);* >>> >>> /** >>> * Releases read lock for a Cluster. This will release the read lock >>> in the following order >>> * 1. for the Cluster >>> * 2. for the Service >>> * >>> * @param serviceName service name to release read lock >>> * @param clusterId cluster id to release read lock >>> */ >>> * public static void releaseReadLockForCluster (String serviceName, >>> String clusterId);* >>> >>> /** >>> * Acquires read lock for the Application >>> * >>> * @param appId Application id >>> */ >>> * public static void acquireReadLockForApplication (String appId) ;* >>> >>> /** >>> * Releases read lock for the Application >>> * >>> * @param appId Application id >>> */ >>> * public static void releaseReadLockForApplication (String appId);* >>> >>> >>> >>> >>> >>> On Mon, Oct 6, 2014 at 2:51 PM, Imesh Gunaratne <im...@apache.org> >>> wrote: >>> >>>> This looks great! As we discussed may be we could provide an interface >>>> in the messaging component to acquire and release locks at different sub >>>> tree levels. The whole idea is to avoid any possibilities of reading the >>>> topology in an inconsistent state. >>>> >>>> Thanks >>>> >>>> On Mon, Oct 6, 2014 at 12:37 PM, Isuru Haththotuwa <isu...@apache.org> >>>> wrote: >>>> >>>>> Thanks Imesh. >>>>> >>>>> I have shown some examples for the new hierarchical locking approach. >>>>> Please do let me know your feedback on this. >>>>> >>>>> Acquire a write lock: >>>>> >>>>> From root level, acquire read lock, and acquire a write lock only >>>>> for the >>>>> relevant sub tree. >>>>> >>>>> Acquire a read lock: >>>>> >>>>> From root level, acquire read locks till the relevant sub tree >>>>> >>>>> Examples - >>>>> >>>>> Example 1: Acquiring write lock for a Cluster to modify the Cluster >>>>> object - >>>>> acquiring: >>>>> 1. acquire read lock for all Services >>>>> 2. acquire read lock for the particular Service, to which >>>>> the cluster belongs to >>>>> 3. acquire write lock for the Cluster >>>>> >>>>> releasing: >>>>> 1. release write lock for the Cluster >>>>> 2. release read lock for the particular Service >>>>> 3. release read lock for all Services >>>>> >>>>> Example 2: Acquiring write lock to add a new Cluster object - >>>>> acquiring: >>>>> 1. acquire read lock for all Services >>>>> 2. acquire write lock for the particular Service, to which >>>>> the cluster belongs to >>>>> >>>>> releasing: >>>>> 1. release write lock for the particular Service >>>>> 2. release read lock for all Services >>>>> >>>>> Example 3: Acquiring read lock to read Cluster information >>>>> acquiring: >>>>> 1. acquire read lock for all Services >>>>> 2. acquire read lock for the particular Service, to which >>>>> the cluster belongs to >>>>> 3. acquire read lock for the relevant Cluster >>>>> >>>>> releasing: >>>>> 1. release read lock for the relevant Cluster >>>>> 2. release read lock for the particular Service >>>>> 3. release read lock for all Services >>>>> >>>>> Example 4: Acquiring the write lock to add a deploy a Cartridge >>>>> (add a new Service) >>>>> acquire: >>>>> 1. acquire write lock for all Services >>>>> >>>>> release: >>>>> 1. release write lock for all Services >>>>> >>>>> In all of these examples, the lock acquiring happens from top of the >>>>> tree to down. This is to avoid deadlocks. >>>>> >>>>> On Mon, Oct 6, 2014 at 10:50 AM, Imesh Gunaratne <im...@apache.org> >>>>> wrote: >>>>> >>>>>> Great! Thanks for the clarification Isuru! >>>>>> >>>>>> Yes I agree, I think what we can do is, identify the sub trees that >>>>>> will not break the consistency of the data structure and manage locks at >>>>>> those sub tree level. >>>>>> >>>>>> Thanks >>>>>> >>>>>> On Sun, Oct 5, 2014 at 8:07 PM, Isuru Haththotuwa <isu...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Hi Lahiru and Imesh, >>>>>>> >>>>>>> Thanks a lot for the input. >>>>>>> >>>>>>> What I do here is locking only the relevant sub tree of the complete >>>>>>> Topology tree, as locking the whole tree is rather inefficient. For an >>>>>>> example, when a MemberActivated event is received, we have the cluster >>>>>>> id >>>>>>> of the cluster that particular member belongs to. IMHO, we only need to >>>>>>> acquire the write lock for that cluster , and do not need the lock for >>>>>>> complete Topology tree. Therefore, any other thread which needs to do >>>>>>> another operation on a separate sub tree (for an example, deploy a new >>>>>>> service, etc.) can do that concurrently. >>>>>>> >>>>>>> On Sun, Oct 5, 2014 at 12:20 AM, Lahiru Sandaruwan <lahi...@wso2.com >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi Isuru, >>>>>>>> >>>>>>>> Looks like a good move to improve the efficiency, >>>>>>>> >>>>>>>> On Fri, Oct 3, 2014 at 3:05 PM, Isuru Haththotuwa < >>>>>>>> isu...@apache.org> wrote: >>>>>>>> >>>>>>>>> I did the initial changes, at the testing phase now. For an >>>>>>>>> example, if we need to add a new Service, we do not need to lock >>>>>>>>> everything >>>>>>>>> now. We an only acquire the write lock on Services, and add the >>>>>>>>> Service. >>>>>>>>> >>>>>>>> >>>>>>>> Service can be an element of a group or an app. So shouldn't we >>>>>>>> lock complete application if we add/modify a service? Otherwise a >>>>>>>> different >>>>>>>> thread might change parents/relationships otherwise. >>>>>>>> >>>>>>> AFAIK a Service gets created when we deploy a cartridge. An >>>>>>> application/Group can refer a service. In the case of modifying an >>>>>>> Application, we do need to lock the relevant clusters that belong to >>>>>>> that >>>>>>> Application. I implemented that. >>>>>>> >>>>>>>> >>>>>>>> So generally i think we can bring down the locking level to >>>>>>>> Application, but not the services. Also if we need to read any part, we >>>>>>>> have to get the read lock for the whole topology, such that the >>>>>>>> receiver >>>>>>>> get a particular snapshot of the topology as Imesh also mentioned. >>>>>>>> >>>>>>> If we need to lock the complete Topology, we can still do that, such >>>>>>> as in a Complete Topology event. But IMHO, if we know the exact part (a >>>>>>> particular Cluster, etc.) that we need to read/write, we do not need to >>>>>>> lock the whole Topology. >>>>>>> >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> If we need to modify a particular Service, can read lock the >>>>>>>>> Services and acquire the write lock on the relevant Service and do the >>>>>>>>> modification. >>>>>>>>> >>>>>>>> This support is there for Services, Cluster and Applications. >>>>>>>>> >>>>>>>> >>>>>>>>> On Wed, Oct 1, 2014 at 9:27 PM, Manula Chathurika Thantriwatte < >>>>>>>>> manu...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Isuru, >>>>>>>>>> >>>>>>>>>> +1 for the hierarchical locking approach. Using hierarchical >>>>>>>>>> locking we can have more benefits [1]. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> http://synapticnulship.com/blog/2013/03/08/comparison-chainlocker-vs-heirarchical-mutexes/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Oct 1, 2014 at 7:46 PM, Isuru Haththotuwa < >>>>>>>>>> isu...@apache.org> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Devs, >>>>>>>>>>> >>>>>>>>>>> In the current Topology implementation, we acquire read/write >>>>>>>>>>> locks on Topology from the root level itself. For an example, if we >>>>>>>>>>> need to >>>>>>>>>>> modify a single Cluster object, we still need to acquire a write >>>>>>>>>>> lock from >>>>>>>>>>> the Topology root level. But, this is a costly operation. >>>>>>>>>>> Specially, with >>>>>>>>>>> Service Grouping changes, we would need to traverse through an >>>>>>>>>>> Application. >>>>>>>>>>> Since an Application can be a recursive structure, it can be a time >>>>>>>>>>> consuming operation. in such a scenario, if we are to lock the whole >>>>>>>>>>> Topology, there will be many threads waiting on that lock. >>>>>>>>>>> >>>>>>>>>>> As a solution, I think we can use hierarchical locking [1]. For >>>>>>>>>>> an example, when you need to obtain the write lock for a particular >>>>>>>>>>> Application, you do not need to lock the whole tree, but can lock >>>>>>>>>>> only that >>>>>>>>>>> Application itself. However, still we need to get the read locks >>>>>>>>>>> for the >>>>>>>>>>> parents. >>>>>>>>>>> >>>>>>>>>>> A separate Lock tree will be maintained for the Topology. >>>>>>>>>>> >>>>>>>>>>> Please share your feedback. >>>>>>>>>>> >>>>>>>>>>> [1]. >>>>>>>>>>> http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/DDJ/2008/0801/071201hs01/071201hs01.html >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Thanks and Regards, >>>>>>>>>>> >>>>>>>>>>> Isuru H. >>>>>>>>>>> +94 716 358 048* <http://wso2.com/>* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Regards, >>>>>>>>>> Manula Chathurika Thantriwatte >>>>>>>>>> Software Engineer >>>>>>>>>> WSO2 Inc. : http://wso2.com >>>>>>>>>> lean . enterprise . middleware >>>>>>>>>> >>>>>>>>>> email : manu...@wso2.com / man...@apache.org >>>>>>>>>> phone : +94 772492511 >>>>>>>>>> blog : http://manulachathurika.blogspot.com/ >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> <http://manulachathurika.blogspot.com/> >>>>>>>>>> Thanks and Regards, >>>>>>>>>> >>>>>>>>>> Isuru H. >>>>>>>>>> <http://manulachathurika.blogspot.com/> >>>>>>>>>> +94 716 358 048 <http://manulachathurika.blogspot.com/>* >>>>>>>>>> <http://wso2.com/>* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> * <http://wso2.com/>* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -- >>>>>>>> Lahiru Sandaruwan >>>>>>>> Committer and PMC member, Apache Stratos, >>>>>>>> Senior Software Engineer, >>>>>>>> WSO2 Inc., http://wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> >>>>>>>> email: lahi...@wso2.com cell: (+94) 773 325 954 >>>>>>>> blog: http://lahiruwrites.blogspot.com/ >>>>>>>> twitter: http://twitter.com/lahirus >>>>>>>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146 >>>>>>>> >>>>>>>> -- >>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> >>>>>>>> Thanks and Regards, >>>>>>>> >>>>>>>> Isuru H. >>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146> >>>>>>>> +94 716 358 048 >>>>>>>> <http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146>* >>>>>>>> <http://wso2.com/>* >>>>>>>> >>>>>>>> >>>>>>>> * <http://wso2.com/>* >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Imesh Gunaratne >>>>>> >>>>>> Technical Lead, WSO2 >>>>>> Committer & PMC Member, Apache Stratos >>>>>> >>>>>> -- >>>>>> Thanks and Regards, >>>>>> >>>>>> Isuru H. >>>>>> +94 716 358 048* <http://wso2.com/>* >>>>>> >>>>>> >>>>>> * <http://wso2.com/>* >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> -- >>>> Imesh Gunaratne >>>> >>>> Technical Lead, WSO2 >>>> Committer & PMC Member, Apache Stratos >>>> >>>> -- >>>> Thanks and Regards, >>>> >>>> Isuru H. >>>> +94 716 358 048 >>>> >>>> -- >>>> <%2B94%20716%20358%20048> >>>> Thanks and Regards, >>>> >>>> Isuru H. >>>> <%2B94%20716%20358%20048> >>>> +94 716 358 048* <http://wso2.com/>* >>>> >>>> >>>> * <http://wso2.com/>* >>>> >>>> >>>> > > > -- > > Gayan Gunarathne > Technical Lead > WSO2 Inc. (http://wso2.com) > email : gay...@wso2.com | mobile : +94 766819985 > > -- > <%2B94%20766819985> > Thanks and Regards, > > Isuru H. > <%2B94%20766819985> > +94 716 358 048 <%2B94%20766819985>* <http://wso2.com/>* > > > * <http://wso2.com/>* > > >
