Hi Zoran, IMM has that cleanTheBasement job which iterates through the whole CCB list every second. That's why this sounds like a performance problem to me.
> I don’t see a valid case where I’m not able to open/use a CCB for > minutes because some other application intensively used CCBs and > managed to close 10000 handles within a minute. That's why we make the limit configurable, to let users increase it when they do something weird. BR, Hung Nguyen - DEK Technologies -------------------------------------------------------------------------------- From: Zoran Milinkovic [email protected] Sent: Monday, September 05, 2016 5:56PM To: Hung Nguyen, Anders Widell, Mahesh Valla, Neelakanta Reddy, Opensaf-devel [email protected], [email protected], [email protected], [email protected], [email protected] Cc: Subject: RE: [devel] Immnd: maximum Ccbs limit 10000 has been reached very quickly Hi Hung, Agree with you and Anders Widell for the first item. For the second item, as I mentioned earlier, IMM is not developed to handle a huge amount of CCBs. If we need to support that, then we need to rewrite CCB part, and replace std::vector with some other structure, possibly with std::map. Then the performance degradation will be minimal. And when this is changed, probably CCB limitation would me more related to memory usage instead of IMM performance. Then we can consider all CCB. To repeat myself again, I don’t see a valid case where I’m not able to open/use a CCB for minutes because some other application intensively used CCBs and managed to close 10000 handles within a minute. BR, Zoran *From:*Hung Nguyen [mailto:[email protected]] *Sent:* den 5 september 2016 12:29 *To:* Anders Widell; Zoran Milinkovic; A V Mahesh; Neelakanta Reddy; [email protected] *Subject:* Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached very quickly Hi, I agree with Anders that we should let users enable the limit instead of enabling it automatically. Maybe we can set default value of those attributes to 0 and IMM will interpret 0 as unlimited. It's going to be a defect ticket for #195. About the second problem (i.e. the meaning of maxCcbs limit), I think the limit should be applied on all CCBs as we store them in the same list. This limit is for pure performance purpose. It should be well documented though. The easiest way for users to check how much resource left is to use 'immadm -O displayverbose'. BR, Hung Nguyen - DEK Technologies -------------------------------------------------------------------------------- From: Anders [email protected] <mailto:[email protected]> Sent: Friday, September 02, 2016 9:53PM To: Zoran Milinkovic, Mahesh Valla, Neelakanta Reddy, Opensaf-devel [email protected] <mailto:[email protected]>,[email protected] <mailto:[email protected]>,[email protected] <mailto:[email protected]>,[email protected] <mailto:[email protected]> Cc: Subject: Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached very quickly We have to consider the backwards compatibility aspect here. The general principle is that if an application works fine with one version of OpenSAF, then it should work fine also with a newer version of OpenSAF. Mahesh has evidently an example application (a test program) that starts to fail after upgrading from OpenSAF 5.0 to OpenSAF 5.1.FC. If it is a pathological example program then you can argue that it is not relevant, but I don't think it is the case here. You could imagine a real-life case where someone has written a shell script that runs the immcfg command in a loop 10000 times. The safest approach to introduce a new limit where we previously didn't have any limit, is to set the default value of the new limit to infinite (unlimited), and let the user take an active decision to configure a lower value of the limit. This way, applications will not get unpleasant surprises after upgrading OpenSAF. regards, Anders Widell On 09/02/2016 10:56 AM, Zoran Milinkovic wrote: -------------------------------------------------------------------------------- From: Zoran Milinkovic [email protected] Sent: Monday, September 05, 2016 5:56PM To: Hung Nguyen, Anders Widell, Mahesh Valla, Neelakanta Reddy, Opensaf-devel [email protected], [email protected], [email protected], [email protected], [email protected] Cc: Subject: RE: [devel] Immnd: maximum Ccbs limit 10000 has been reached very quickly Hi Mahesh, Ticket #195 introduced this limitation. After all, I'm not sure who is right... Neelakanta or me :) Earlier, we didn't have any limitation, and limitation is introduced to have some reasonable amount of CCB handles in IMM. IMM keeps non-used and active CCB handles in the same place. If the amount of CCB handles grows a lot, then we will see a degradation of IMM performance. >From earlier discussion with Anders, it's not meant from the beginning that CCB will be used so often. This might be very specific case. That's also the reason why CCB handles are handled in a way that is not good for huge amount of CCB handles. >From this point of view, Neelakanta is correct. >From my point of view, we should only consider CCBs that can be used. I don't see acceptable that CCB operations cannot be done for minutes if we have a huge amount of closed CCBs. We can improve CCB handling in the next OpenSAF release. Now it's too late. For this particular problem, I would go with checking only CCBs that can be used (CCB that have SA_AIS_OK on mVeto variable, or calling ->isOk() method on CcbInfo struct). This solution may have impact on IMM performance. Or we can revert the ticket and implement it in the next release with better performance. I would also like to hear Neelakanta's and Hung's proposals. Thanks, Zoran -----Original Message----- From: A V Mahesh [mailto:[email protected]] Sent: den 2 september 2016 05:56 To: Zoran Milinkovic; Neelakanta Reddy;[email protected] <mailto:[email protected]> Subject: Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached very quickly Hi Zoran, Thanks for the update , you are right ERR_NO_RESOURCES only justifiable if 10000 IMM application concurrently holding 10000 CCB`s ( active CCB), in current case agent is already finalized. By the way by roll backing which change set I can continue my testing ? -AVM On 9/1/2016 5:52 PM, Zoran Milinkovic wrote: Hi Neelakanta, This is definitely a bug. When CCB is finalized, it's still shown as an active CCB. If you check number of CCBs with resource display functionality, you will see that number of CCBs is not decreasing. BR, Zoran -----Original Message----- From: Neelakanta Reddy [mailto:[email protected]] Sent: den 1 september 2016 11:09 To:[email protected] <mailto:[email protected]> Subject: Re: [devel] Immnd: maximum Ccbs limit 10000 has been reached very quickly Hi, I think this is not yet documented and will be documented. Try to increase the ccb limits and check. /Neel. On 2016/09/01 02:01 PM, Chani Srivastava wrote: Hi, With current 5.1 OpenSAF changeset 7997 the issue is reproducible ============================================== Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10002 COMMITTED (chaniTestClass) Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10003 COMMITTED (chaniTestClass) Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10004 COMMITTED (chaniTestClass) Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO Ccb 10005 COMMITTED (chaniTestClass) Sep 1 12:16:52 OSAF-SC1 osafimmnd[27298]: NO ERR_NO_RESOURCES: maximum Ccbs limit 10000 has been reached for the cluster =============================================== This looks like a candidate for ticket. Let me know i'll raise it in community. -Chani On 9/1/2016 1:49 PM, Neelakanta Reddy wrote: Hi, can you verify the same with 5.1. /Neel. On 2016/09/01 12:58 PM, Chani Srivastava wrote: Hi All, I verified this is a break in functionality. In OpenSAF 5.0 I tried creating 15000 objects per CCB and it worked fine. ===================================== Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 237 COMMITTED (chaniTestClass) Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 238 COMMITTED (chaniTestClass) Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 239 COMMITTED (chaniTestClass) Nov 26 15:21:17 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 240 COMMITTED (chaniTestClass) . . . Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 15230 COMMITTED (chaniTestClass) Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 15231 COMMITTED (chaniTestClass) Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 15232 COMMITTED (chaniTestClass) Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 15233 COMMITTED (chaniTestClass) Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 15234 COMMITTED (chaniTestClass) Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 15235 COMMITTED (chaniTestClass) Nov 26 15:23:05 SCALE_SLOT-91 osafimmnd[24451]: NO Ccb 15236 COMMITTED (chaniTestClass) ====================================== The ticket #195 only makes the MAX parameters configurable. If accepted, I'll raise the ticket in sourceforge. -Chani On 9/1/2016 10:59 AM, A V Mahesh wrote: Hi All, I was running `immcfg` in a loop to create some object , once it reaches 10001objects creation Immnd is returning `ERR_NO_RESOURCES: maximum Ccbs limit 10000 has been reached for the cluster` error which is unexpected , once `immcfg` reruns , it is expected that the `maximum Ccbs` will decremented ( this was previous behavior) , can you please check. ================================================================== = ====================================== for (( i = 1 ; i <=300000; i++)) immcfg -c PinvId -a pinvPhoneNumber=+46768 pinvRdn=$i Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 9996 COMMITTED (immcfg_SC-1_2334) Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 9997 COMMITTED (immcfg_SC-1_2337) Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 9998 COMMITTED (immcfg_SC-1_2340) Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 9999 COMMITTED (immcfg_SC-1_2343) Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 10000 COMMITTED (immcfg_SC-1_2346) Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO Ccb 10001 COMMITTED (immcfg_SC-1_2349) Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO ERR_NO_RESOURCES: maximum Ccbs limit 10000 has been reached for the cluster Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO ERR_NO_RESOURCES: maximum Ccbs limit 10000 has been reached for the cluster Sep 1 10:43:33 SC-1 osafimmnd[4466]: NO ERR_NO_RESOURCES: maximum Ccbs limit 10000 has been reached for the cluster ================================================================== = ====================================== -AVM ------------------------------------------------------------------ - ----------- _______________________________________________ Opensaf-devel mailing list [email protected] <mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/opensaf-devel ------------------------------------------------------------------- - ---------- _______________________________________________ Opensaf-devel mailing list [email protected] <mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/opensaf-devel -------------------------------------------------------------------- - --------- _______________________________________________ Opensaf-devel mailing list [email protected] <mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/opensaf-devel --------------------------------------------------------------------- - -------- _______________________________________________ Opensaf-devel mailing list [email protected] <mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/opensaf-devel ---------------------------------------------------------------------- -------- _______________________________________________ Opensaf-devel mailing list [email protected] <mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/opensaf-devel ---------------------------------------------------------------------- -------- _______________________________________________ Opensaf-devel mailing list [email protected] <mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/opensaf-devel ------------------------------------------------------------------------------ _______________________________________________ Opensaf-devel mailing list [email protected] <mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/opensaf-devel ------------------------------------------------------------------------------ _______________________________________________ Opensaf-devel mailing list [email protected] <mailto:[email protected]> https://lists.sourceforge.net/lists/listinfo/opensaf-devel ------------------------------------------------------------------------------ _______________________________________________ Opensaf-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/opensaf-devel
