On Tue, Jun 7, 2011 at 3:02 PM, Tharindu Mathew <thari...@wso2.com> wrote:
> For high load systems, we can always recommend to increase the event > threshold. The practical approach is to define a maximum queue length for executor. If the system hit this limit system drop the messages effectively increasing the threshold automatically. If events are published within a time interval, that will work with stat module since there is no accumulation of memory. But if the events has to store in memory (for activity monitoring) then the amount of memory need will grow with increase of message rate. Again the only solution is to limit the size and drop if goes after some point. Therefore first, the data saving path of the BAM publishers should be optimized so that a rate at which it can work goes to the possible upper limit. After that there should be a proper mathematical model which make it operate at the highest point in a high load and drop messages after that without going to OOM. thanks, Amila. > > When bursts occur, the default value of 20, is definitely not enough. > > > On Tue, Jun 7, 2011 at 2:47 PM, Amila Suriarachchi <am...@wso2.com> wrote: > >> In a separate note the BAM publisher memory leak is not related to Qpid >> memory leak which we experience in long running. >> >> BAM publisher problem is mainly due to speed mismatching. i.e BAM >> publisher publish the messages to its internal thread pool queue in a much >> higher rate than it submit the message to the Qpid queue. This is becuase it >> take some time to publish the message to Qpid and it uses only one thread >> for that. (even with multiple threads it failing) >> >> In other words this OOM issue is caused by the tasks accumilated in the >> BAM thread pool. >> >> The only solution for this is to improve the performace of the path which >> saves the BAM messages to the database. >> >> Directly saving messages to the data base using jdbc from the BAM >> publisher it self is the fastest and hence can withhold for higher speeds. >> >> Anyway even that has an upper limit after which there will be an OOM error >> due to speed mismatch. >> >> thanks, >> Amila. >> >> On Mon, Jun 6, 2011 at 1:28 PM, Sumedha Rubasinghe <sume...@wso2.com>wrote: >> >>> This is the conclusion from the discussion happened little while ago. >>> >>> As Amila mentioned, we can switch to in-memory storage from Qpid. But >>> this has not been fully test. We will test this set up (in-memory) using BAM >>> & then make the change permanent for 3.2.0. >>> >>> This way, each of the products will have a minimum impact from a code >>> change perspective. >>> >>> /sumedha >>> >>> >>> On Mon, Jun 6, 2011 at 10:50 AM, Tharindu Mathew <thari...@wso2.com>wrote: >>> >>>> >>>> >>>> On Mon, Jun 6, 2011 at 8:36 AM, Sanjiva Weerawarana >>>> <sanj...@wso2.com>wrote: >>>> >>>>> On Sun, Jun 5, 2011 at 5:11 PM, Tharindu Mathew <thari...@wso2.com>wrote: >>>>> >>>>>> That can be achieved by using a simple non-blocking Axis2 sender type >>>>>>> thing - which I believe is what Hiranya wrote a while ago. >>>>>>> >>>>>>> The BAM server needs to be set up in a clustered manner with an LB >>>>>>> etc. to handle the large number of messages it will receive when >>>>>>> Stratos is >>>>>>> running at full tilt (and have additional load balancing things like DNS >>>>>>> round-robin). We can make that work easily. >>>>>>> >>>>>>> +1, for this model. The publisher that is used in Stratos is already >>>>>> done in a non blocking way. I feel it's better to go for the jdbc model. >>>>>> >>>>> >>>>> Tharindu what is the "JDBC model"?? We don't need persistence for these >>>>> messages, so where does JDBC come into the picture? >>>>> >>>> >>>> In the publisher, Hiranya wrote, we directly inject the event into the >>>> 'BAM' DB. This is what I referred to as the jdbc model. Instead of sending >>>> an event to BAM, and then inserting to the BAM DB. This option is also >>>> provided, which is what we use by default. >>>> >>>>> >>>>> Sanjiva. >>>>> -- >>>>> Sanjiva Weerawarana, Ph.D. >>>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>>> email: sanj...@wso2.com; phone: +94 11 763 9614; cell: +94 77 787 6880| +1 >>>>> 650 265 8311 >>>>> blog: http://sanjiva.weerawarana.org/ >>>>> >>>>> Lean . Enterprise . Middleware >>>>> >>>>> _______________________________________________ >>>>> Carbon-dev mailing list >>>>> Carbon-dev@wso2.org >>>>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Tharindu >>>> >>>> >>>> _______________________________________________ >>>> Carbon-dev mailing list >>>> Carbon-dev@wso2.org >>>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>>> >>>> >>> >>> _______________________________________________ >>> Carbon-dev mailing list >>> Carbon-dev@wso2.org >>> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >>> >>> >> >> _______________________________________________ >> Carbon-dev mailing list >> Carbon-dev@wso2.org >> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >> >> > > > -- > Regards, > > Tharindu > > > _______________________________________________ > Carbon-dev mailing list > Carbon-dev@wso2.org > http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev > >
_______________________________________________ Carbon-dev mailing list Carbon-dev@wso2.org http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
