Thread dump for reference https://drive.google.com/a/wso2.com/file/d/0B92QGepCpPD9SFdjQi1OeV9DUUk/view?usp=sharing
Thanks *,Sajith Ravindra* Senior Software Engineer WSO2 Inc.; http://wso2.com lean.enterprise.middleware mobile: +94 77 2273550 blog: http://sajithr.blogspot.com/ <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> On Thu, Jun 16, 2016 at 2:18 PM, Sajith Ravindra <[email protected]> wrote: > Hi all, > > We are doing a performance test on IS analytics server. While we keep > pumping data to the server we can see that throughput gradually decreases > to a very low (< 1K) TPS. We used MySQL with InnoDB with tuned parameters. > And set Xms2g and Xmx4g in DAS. > > When looked at the thread dumps we can see that more than 100 threads > BLOCKED/TIMED_WATING at AnalyticsEventQueueManager.put() on a single lock > object(i.e. on the same reference). Many threads are just waiting to get > the lock without processing events which seem to be the reason for degraded > performance. > > Are there any parameters that we can tune to gain better performance to > overcome this bottleneck ? > > I had a look at the code and AFAIU all events of a particular tenant are > persisted through a single synchronized queue. According to my > understanding, this synchronized queue is a performance bottleneck since > all threads pertaining to a persisted event stream will need to block on > this queue. > > IMO if we can have a queue per event-sink(i.e per persisted stream) it > should give a better performance than this. Any implications if we do that > or is there a better option? > > > Thanks > *,Sajith Ravindra* > Senior Software Engineer > WSO2 Inc.; http://wso2.com > lean.enterprise.middleware > > mobile: +94 77 2273550 > blog: http://sajithr.blogspot.com/ > <http://lk.linkedin.com/pub/shani-ranasinghe/34/111/ab> >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
