Hi Johann, I have avoid creating role for each tmp queue when subscription adding to non durable topic. Now we only create one internal role by topic name and assign subscribe, publish and changePermission. Please find the PR [1]. We have build a MB 3.0.0-SNAPSHOT pack with all latest changes including PR [1] and check again with MSSQL. But we still encounter same dead-lock issue. We are only adding 5 subscription at once. We have 2 node cluster setup. Just add 2 subscription to node1 and 3 subscription to node2. It seems root cause is not because of creating tmp role for each subscription. Now we are adding patch given by Indunil, and check whether it resolve the issue and update mail.
[1] https://github.com/wso2/carbon-business-messaging/pull/170 Cheers! On Fri, Oct 2, 2015 at 5:08 PM, Johann Nallathamby <joh...@wso2.com> wrote: > Hi All, > > We had a discussion involving MB team members and IS team members. > > Summary of the discussion: > > Possibility of deadlocks occurring have drastically reduced in MySQL and > Oracle with the patches given so far and tested with high levels of > concurrency. Only MSSQL gives deadlocks and that too for very small > concurrency levels. In our experience we have in fact encountered several > deadlocking issues with MSSQL setups previously also in certain other > scenarios, and had to move to a non DB solution. > > At this moment we have fixed the code level issues which we have found so > far. Beyond this it may be hard to find issues at code level. The fixes > done so far also, doesn't completely eliminate the possibility of a > deadlock; they only reduce it to a large extent, but that level doesn't > seem to be enough for MSSQL. From our side we will look into profiling this > use case on MSSQL server. > > If we are to completely eliminate deadlocking we will have to lock the > tables which we haven't done as a practice because it will drastically > reduce performance. Other option available is for the MB code to recover > from the specific exception by retrying. > > However during the discussion there were several problems raised regarding > implementation of topics. > > MB uses internal roles and permissions in user kernel to authorize access > to Topics and Queues. > > First we figured out that permission (resource + action) are being created > for each subscription (queues or topics). And for each subscription an > internal role is also being created. The creation of permission seems > redundant, because the resource name actually has a one-to-one > correspondence with the role name (resource name can be derived from role > name), and action can contain only 3 values, publish, subscribe and > changePermission. Every time a new subscription is added a new role and 3 > new permissions are added for that subscription which is unwanted overhead. > Every time a subscription is deleted the corresponding role and 3 > permissions are deleted. In a high load scenario there are many DB calls > going on which effects performance drastically. > > In this case what we discussed with Indika is that, the permissions can be > fixed to publish, subscribe and changePermission; they can be added > through component.xml and will appear in the permission tree as well. Only > the internal role can be created and those 3 permissions assigned. This way > we are adding and deleting the internal role only and not doing any > operations to the UM_PERMISSION table which will reduce a lot of DB calls. > > The problem happens only for topics and not for queues, because in queues > you don't seem to be creating roles or permissions per subscription. But > for topics you do that and that seems to be executing huge amount of DB > calls on the user-mgt tables to add and remove roles and permissions. > > Further when we discussed we also pointed out that creating an internal > role per topic subscription is an unwanted overhead. Normally we create > roles if we are going to assign a set of users to it. Using internal roles > for queues seems correct. The use case requires users other than the > subscriber to be added to the role for that topic. In case of topics, as > per the implementation, you create a role per subscription (internally > create a temp queue) and and assign the subscriber to it. As per Indika and > Hasitha there is no use case to assign other users to it. The only > requirement is to keep track of the user who subscribed to that temp queue. > In this case creating a role as well as 3 permissions for it is > unnecessary. You can just keep the name of the user along with the > subscription/temp queue name in the metadata table that MB has. > > Please look into these options. As per IS team's understanding > improvements along these lines will give a substantial improvement. We can > help you in implementing these changes. > > Regards, > Johann. > > > > On Thu, Oct 1, 2015 at 2:36 PM, Manuri Amaya Perera <manu...@wso2.com> > wrote: > >> Hi Indunil, >> >> As I understand, "FOREIGN KEY constraint violation" issue is a >> regression issue therefore the deadlock issue cannot be resolved until >> foreign key constraint violation issue is fixed, right? >> >> Thank you. >> >> On Thu, Oct 1, 2015 at 2:01 PM, Indunil Upeksha Rathnayake < >> indu...@wso2.com> wrote: >> >>> Hi, >>> The deadlock issue in https://wso2.org/jira/browse/MB-1326 is resolved >>> in both oracle 12c and 11g r2. (Please refer previous mails in this mail >>> thread to get the idea of the issue). Then after a "FOREIGN KEY constraint >>> violation" issue occurred in both mssql and oracle, intermittently (Refer >>> the attached file "errorLog_foriegnKeyViolation" for the error log). For >>> that, issued the fix in https://github.com/wso2/carbon-kernel/pull/512, >>> and it worked for oracle, but not for mssql. This is tested in following >>> scenarios in MB cluster. >>> >>> 1. 5 queues - 5 subscribers and 5 consumers ( 1 publisher and 1 consumer >>> each ) >>> 2. 5 Topics - 5 subscribers and 5 consumers ( in mssql, it breaks for >>> this scenario) >>> 3. 100 topics - 100 publishers and 100 consumers ( 1 publisher and 1 >>> consumer each ) >>> 4. 1 topic - 100 subscribers / 100 consumers >>> 5. Mixed scenario where multiple queues, topics and durable topics are >>> used >>> >>> I have attached all the debug logs here. Please consider this matter and >>> provide your feedbacks on resolving this. >>> >>> Thanks and Regards >>> -- >>> Indunil Upeksha Rathnayake >>> Software Engineer | WSO2 Inc >>> Email indu...@wso2.com >>> >>> >> >> >> -- >> >> *Manuri Amaya Perera* >> >> *Software Engineer* >> >> *WSO2 Inc.* >> >> *Blog: http://manuriamayaperera.blogspot.com >> <http://manuriamayaperera.blogspot.com>* >> > > > > -- > Thanks & Regards, > > *Johann Dilantha Nallathamby* > Technical Lead & Product Lead of WSO2 Identity Server > Governance Technologies Team > WSO2, Inc. > lean.enterprise.middleware > > Mobile - *+94777776950* > Blog - *http://nallaa.wordpress.com <http://nallaa.wordpress.com>* > -- Indika Sampath Senior Software Engineer WSO2 Inc. http://wso2.com Phone: +94 716 424 744 Blog: http://indikasampath.blogspot.com/
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
