On Sun, Jul 31, 2016 at 12:59 PM, Malith Jayasinghe <mali...@wso2.com> wrote:
> > > On Sun, Jul 31, 2016 at 12:49 PM, Ishara Karunarathna <isha...@wso2.com> > wrote: > >> Hi Malith, >> >> On Sun, Jul 31, 2016 at 12:37 PM, Malith Jayasinghe <mali...@wso2.com> >> wrote: >> >>> HI Indunil, >>> Just a few question regarding this performance test you have done: >>> >>> What is the reason for selecting the concurrency = 500 here? >>> >> This is the user expected concurrency level. Thats the reason we user >> this. >> >>> >>> Have you tested the behaviour for lower concurrency levels? >>> >>> *"currently the TPS is dropping from the initial TPS 1139.5/s to 198.1/s >>> in around 6100000 user count.(User Add)" - *How did you notice/measure >>> this drop in TPS? Did you analyze the jmeter results offline? After it >>> drops, does it improve after some time or does it stay the same? >>> >> We test this with the Jmeter summery report. >> with latest results if we start again few min (2min) we can get this max >> tps and come down to around 250tps >> > > Ok so it comes down to 250tps and stays there? Are you running these > tests without a warm-up period? > Nop. With 2s worm up and then 10s ramp up period > >>> >>> Did you look at the behaviour of latency? >>> >>> Thanks >>> >>> Malith >>> >>> >>> On Fri, Jul 29, 2016 at 2:57 PM, Indunil Upeksha Rathnayake < >>> indu...@wso2.com> wrote: >>> >>>> Hi, >>>> >>>> We are currently engaged into a performance analysis where we are >>>> analyzing performance for User Add, Update, Authentication operations. The >>>> testing has been carried out in a following environment with 500 >>>> concurrency and users up to 10 million. >>>> >>>> *Environment :* >>>> >>>> m3.2xlarge ( 8 core, 30GB, SSD 2x80 GB) 3 instances. >>>> MySQL 5.7 >>>> Ubuntu 14.04 >>>> Openldap-2.4.31 >>>> IS 5.1.0 >>>> >>>> In order to optimize the MYSQL server, following server parameters have >>>> been tuned accordingly. We have referred MYSQL documentation [1] as well as >>>> have performed analysis using several MYSQL tuners in [2]. >>>> >>>> (1) *max_connections : 1000* (The maximum permitted number of >>>> simultaneous client connections.) >>>> >>>> (2) *join_buffer_size : 259968* (The minimum size of the buffer that >>>> is used for plain index scans, range index scans, and joins that do not use >>>> indexes and thus perform full table scans.) >>>> >>>> (3) *innodb_buffer_pool_size : 5207959552 <5207959552>* (size of the >>>> memory area where InnoDB caches table and index data) >>>> >>>> (4) *innodb_log_buffer_size : 16777216* (size of the buffer for >>>> transactions that have not been committed yet) >>>> >>>> (5) *innodb_buffer_pool_instances : 1* (The number of buffer pool >>>> instances. According to the mysql documentation[1], on systems with a large >>>> amount of memory, we can improve concurrency by dividing the buffer pool >>>> into multiple buffer pool instances. But couldn't change since it's a read >>>> only variable) >>>> >>>> (6) *key_buffer_size : 384000000* (size of the buffer used for index >>>> blocks) >>>> >>>> (7) *table_open_cache : 4000* (The number of open tables for all >>>> threads) >>>> >>>> (8) *sort_buffer_size : 4000000* (Each session that must perform a >>>> sort allocates a buffer of this size) >>>> >>>> (9) *read_buffer_size : 1000000* (Each thread that does a sequential >>>> scan for a table allocates a buffer of this size for each table it scans. >>>> If we do many sequential scans, we might want to increase this value) >>>> >>>> (10) *query_cache_type : 0 * >>>> >>>> (11) *query_cache_limit : 1048576* (Do not cache results that are >>>> larger than this number of bytes) >>>> >>>> (12) *query_cache_size : 1048576* (The amount of memory allocated for >>>> caching query results) >>>> >>>> (13) *thread_stack : 262144* (The stack size for each thread) >>>> >>>> (14) *net_buffer_length : 16384* (Each client thread is associated >>>> with a connection buffer and result buffer. Both begin with a size given by >>>> net_buffer_length but are dynamically enlarged up to max_allowed_packet >>>> bytes as needed) >>>> >>>> (15) *max_allowed_packet : 4194304* (The maximum size of one packet or >>>> any generated/intermediate string) >>>> >>>> (16) *thread_cache_size : 30* (no of threads the server should cache >>>> for reuse) >>>> >>>> >>>> >>>> IS has been configured as follows to optimize the performance. >>>> >>>> (1) JVM Heap Settings (-Xms -Xmx) changed as follows: >>>> >>>> *Xms : 2g * >>>> >>>> *Xmx : 2g * >>>> >>>> (2) Removed following entry from >>>> <IS_HOME>/repository/conf/tomcat/catalina-server.xml to disable http access >>>> logs. >>>> >>>> <Valve className="org.apache.catalina.valves.AccessLogValve" >>>> directory="${carbon.home}/repository/logs" prefix="http_access_" >>>> suffix=".log" pattern="combined" /> >>>> >>>> (3) Tuned following parameters in axis2client.xml file. >>>> >>>> <parameter name="*defaultMaxConnPerHost*">1000</parameter> >>>> >>>> <parameter name="*maxTotalConnections*">30000</parameter> >>>> >>>> (4) Added following additional parameters to optimize database >>>> connection pool. >>>> >>>> <Property name="*maxWait*">60000</Property> >>>> >>>> <Property name="*maxActive*">600</Property> >>>> >>>> <Property name="*initialSize*">20</Property> >>>> >>>> (5) Tuning Tomcat parameters in >>>> <IS_HOME>/repository/conf/tomcat/catalina-server.xml. >>>> >>>> *acceptorThreadCount = 8 * >>>> >>>> *maxThreads="750" * >>>> >>>> *minSpareThreads="150" * >>>> >>>> *maxKeepAliveRequests="600" * >>>> >>>> *acceptCount="600"* >>>> >>>> >>>> >>>> JMeter has been configured as follows to optimize the performance. >>>> >>>> (1) JVM Heap Settings (-Xms -Xmx) changed as follows: >>>> >>>> *Xms : 1g * >>>> >>>> *Xmx : 1g * >>>> >>>> >>>> We were able to optimize the environment up to some level. But* >>>> currently the TPS is dropping from the initial TPS 1139.5/s to 198.1/s in >>>> around 6100000 user count.(User Add)* >>>> >>>> Appreciate your help on figuring out whether we need to do any >>>> modifications to the optimizations in MYSQL, IS and JMeter servers or to >>>> identify the exact issue for this sudden TPS dropping. >>>> >>>> [1] http://dev.mysql.com/doc/refman/5.7/en/optimizing-server.html >>>> >>>> [2] http://www.askapache.com/mysql/mysql-performance-tuning.html >>>> >>>> >>>> Thanks and Regards >>>> -- >>>> Indunil Upeksha Rathnayake >>>> Software Engineer | WSO2 Inc >>>> Email indu...@wso2.com >>>> Mobile 0772182255 >>>> >>> >>> >>> >>> -- >>> Malith Jayasinghe >>> >>> >>> WSO2, Inc. (http://wso2.com) >>> Email : mali...@wso2.com >>> Mobile : 0770704040 >>> Lean . Enterprise . Middleware >>> >> >> >> >> -- >> Ishara Karunarathna >> Associate Technical Lead >> WSO2 Inc. - lean . enterprise . middleware | wso2.com >> >> email: isha...@wso2.com, blog: isharaaruna.blogspot.com, mobile: >> +94717996791 >> >> >> > > > -- > Malith Jayasinghe Ph.D. > Director > > > WSO2, Inc. (http://wso2.com) > Email : mali...@wso2.com > Mobile : 0770704040 > Lean . Enterprise . Middleware > -- Ishara Karunarathna Associate Technical Lead WSO2 Inc. - lean . enterprise . middleware | wso2.com email: isha...@wso2.com, blog: isharaaruna.blogspot.com, mobile: +94717996791
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev