Correction, we are using Apache ISIS 1.11.1 Cesar.
On Tue, 2016-05-31 at 18:54 +0000, César Camilo Lugo Marcos wrote: > Hello Dan. Answers to your questions: > > - which version of Isis are you on? > 1.12.0 > - how have you implemented the stress test? is it automated, eg JMeter, or > just adhoc manual testing? > Using JMeter. We recorded a quite simple script with about 6 read only > operations using th Wicket viewer. All samplers are the recorded HTTP. > - is this stress testing the app running locally on Tomcat, or up in the > cloud (on Amazon Elastic Beanstalk)? > We have it installed in the AWS Elastic Beanstalk with PostgreSQL. > We also made a test on a local host with Tomcat and MySQL, obtaining very > similar results. > > - how much DB traffic does a single request create? Is it all read-only; > > or are there also writes? > Quite low traffic, About 6 read only operations per user. > - for reads, are you seeing any N+1 issues? if so, have you tried to prevent > them using DataNucleus eager loading hint (the "fetch groups" concept)? > Not sure what N+1 is. We are letting Datanucleous handle all foreign keys > and primary keys. We have not configured any fetch groups or any other > configurations than defaults. > - for writes, are they expected? > Did not understand this question. Please explain. > - have you configured any of the various addons that could be writing to the > DB, eg auditing, command or publishing? are these perhaps enabled for > safe/query-only actions? If so for any looping within the app itself, can > you eliminate using the QueryResultsCache? > We are only using the logging add-on, to log login and logout operations. > No auditing, command or publishing add ons. > - what data volumes is this on? If large volumes of data, are there perhaps > missing indices? What's the performance like with small volumes of data? > No Data Object or table has more that a few tens of records. > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? What's the > performance like of one versus the other > This is PostgreSQL. Also tried MySQL with very similar results. Have not > tried in memory HSQLDB. > > > Cesar. > > > On Tue, 2016-05-31 at 18:53 +0100, Dan Haywood wrote: > > > Hi Cesar, > > Those action semantics don't play a large part in the transaction > management; they are mostly used to determine which type of HTTP method to > expose the action under (GET, PUT or POST) and there is also the constraint > that contributed/mixin properties/collections can only be supplied by SAFE > actions. The _ARE_YOU_SURE variants only impact the Wicket viewer. The > _REQUEST_CACHEABLE is only honoured if the action is invoked via the > wrapper factory. > > As Martin and Jeroen have said, using JProfiler or similar will likely > yield valuable info. > > If the issue is with database concurrency (which does seem like a good > hypothesis), then for the most part that is under the management of > DataNucleus itself. There are a bunch of configuration properties that can > be specified in persistor_datanucleus.properties (or isis.properties, > actually); we strip off any "isis.persistor.datanucleus.impl" prefix and > pass the rest of the key down to DN to configure. > > Questions for you: > - which version of Isis are you on? > - how have you implemented the stress test? is it automated, eg JMeter, or > just adhoc manual testing? > - is this stress testing the app running locally on Tomcat, or up in the > cloud (on Amazon Elastic Beanstalk)? > - how much DB traffic does a single request create? Is it all read-only; > or are there also writes? > - for reads, are you seeing any N+1 issues? if so, have you tried to > prevent them using DataNucleus eager loading hint (the "fetch groups" > concept)? > - for writes, are they expected? > - have you configured any of the various addons that could be writing to > the DB, eg auditing, command or publishing? are these perhaps enabled for > safe/query-only actions? If so > - for any looping within the app itself, can you eliminate using the > QueryResultsCache? > - what data volumes is this on? If large volumes of data, are there > perhaps missing indices? What's the performance like with small volumes of > data? > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? What's > the performance like of one versus the other > > I think what I would look for is fast response times for a 1-user system > running locally on Tomcat, with an inmemory database, and low data > volumes. From there, ramp up in different ways, but change one thing at a > time. > > HTH > Dan > > > > On 31 May 2016 at 15:12, César Camilo Lugo Marcos > <cesar.l...@sisorg.com.mx<mailto:cesar.l...@sisorg.com.mx>> > wrote: > > > Thank you both for the hint. I am looking into JProfiler. Still, > > anything about the concurrency control options? > > > > Cesar. > > > > On Tue, 2016-05-31 at 11:13 +0200, Jeroen van der Wal wrote: > > > I agree with Martin that profiling is the only way to go. To illustrate: > > we > > > recently made some code 8 times faster by a few simple code changes on > > > bottlenecks revealed by JProfiler. And those were in places that we've > > > never guessed. > > > > > > On 31 May 2016 at 08:39, Martin Grigorov > > > <mgrigo...@apache.org<mailto:mgrigo...@apache.org>> wrote: > > > > > > > Hi, > > > > > > > > To find out where is the problem you will have to use a profiler like > > > > JProfiler, Yourkit, JVisualVM, etc. > > > > Even some thread dumps would help to see what is going on. > > > > > > > > Martin Grigorov > > > > Wicket Training and Consulting > > > > https://twitter.com/mtgrigorov > > > > > > > > On Mon, May 30, 2016 at 9:00 PM, César Camilo Lugo Marcos < > > > > cesar.l...@sisorg.com.mx<mailto:cesar.l...@sisorg.com.mx>> wrote: > > > > > > > > > Hello, > > > > > > > > > > I would like to add to this topic the following: > > > > > > > > > > Most of the transactions we are testing in these stress tests are not > > > > > bound in ACTIONS, but just plain reads or default transactions using > > > > > Apache ISIS wicket viewer defaults. I don't see any place where I > > could > > > > > define semantics for default read or write transactions not bound > > into > > > > > ACTION methods. Is there any place I should look into for that? > > > > > > > > > > Cesar. > > > > > > > > > > > > > > > On Mon, 2016-05-30 at 18:45 +0000, César Camilo Lugo Marcos wrote: > > > > > > Hello, > > > > > > > > > > > > We have sarted performing some stress tests to our Apache ISIS > > > > > > application. We have found this behavior: > > > > > > > > > > > > - If we run 1 concurrent user, average response times to the main > > > > object > > > > > > reads through the wicket viewer are from 1 to 1.5 seconds. > > > > > > - If we run 2 concurrent users, same transactions go to average > > > > response > > > > > > times up to 16 to 27 seconds. > > > > > > - If we run 10 concurrent users, the transactions start to slow > > down > > > > > > significantly until the app server freezes and we have to restart > > it. > > > > > > > > > > > > As you can see, this is a very significant increase in response > > time > > > > for > > > > > > such a slight change in user load (from 1 user to 2 users). So we > > are > > > > > > guessing we should look into the concurrency control. > > > > > > > > > > > > In the documentation I have found that the only way to influence > > the > > > > way > > > > > > Apache ISIS manages transactions and concurrency checking is by > > using > > > > > > the semantics configuration of the ACTION annotation. > > > > > > > > > > > > semantics=SAFE_AND_REQUEST_CACHEABLE > > > > > > semantics=SAFE > > > > > > semantics=IDEMPOTENT > > > > > > semantics=IDEMPOTENT_ARE_YOU_SURE > > > > > > semantics=NON_IDEMPOTENT > > > > > > semantics=NON_IDEMPOTENT_ARE_YOU_SURE > > > > > > > > > > > > I just wanted to confirm if this is the place to look into, or are > > > > there > > > > > > any other places where we should be looking into too, to solve the > > > > > > performance issue. > > > > > > > > > > > > Cesar. > > > > > > > > > > > > > > > > > > > >
