Glad that seems to have addressed the issue; the fix will be in the 1.13.0 which I'm hoping will be released in the next 2 weeks.
Regarding [3], I did complete the review of any other use of 'synchronized'. In most places the keyword seemed to be unjustified so has been removed, but I don't expect that those removals will cause any appreciable improvements in throughput. -- Dan On 6 June 2016 at 23:53, César Camilo Lugo Marcos <cesar.l...@sisorg.com.mx> wrote: > Dan, > > In our first tests we experienced a very significant improvement. Before > 2 concurrent users significantly decreased response time, while 5 > friezed the server. Now we have ran 100 and 400 concurrent users, and > response time does not get impacted that much. Very significant > improvement, definitely looks like you are heading in the right > direction. Any updates on this will be appreciated as per [3]. > > Cesar. > > On Sat, 2016-06-04 at 11:56 +0100, Dan Haywood wrote: > > A further update re [1]; > > > > I've now pushed a commit [2] which seems to have addressed Cesar's issue > > ... removing a synchronized modifier. > > > > I'm going to do a review of other uses of 'synchronized', to see if they > > are justified or not [3]. > > > > Thx > > Dan > > > > > > [1] https://issues.apache.org/jira/browse/ISIS-1421 > > [2] > > > https://github.com/apache/isis/commit/746176991b523fe9e8ab069392d83ff17af08624 > > [3] https://issues.apache.org/jira/browse/ISIS-1428 > > > > > > > > > > On 3 June 2016 at 13:28, Dan Haywood <d...@haywood-associates.co.uk> > wrote: > > > > > Just as an update on this; I've run Cesar's JMeter tests, and it has > > > revealed an issue in Isis core ... my suspicion being a possible > > > synchronized deadlock. [1] > > > > > > Will investigate with a view to fixing in 1.13.0 > > > > > > Thx > > > Dan > > > > > > > > > [1] https://issues.apache.org/jira/browse/ISIS-1421 > > > > > > On 2 June 2016 at 16:50, César Camilo Lugo Marcos < > > > cesar.l...@sisorg.com.mx> wrote: > > > > > >> Hello. > > >> > > >> We are looking into several recommended topics to tune up the > > >> application. One of them is the use of JDO Optimistic locking instead > > >> of the Datanucleus default pessimistic locking, to improve > concurrency. > > >> I have not found a setting within Datanucleus to change this setting > > >> globally or per transaction, just found this to define the property to > > >> be used as version on the domain object: > > >> > > >> @PersistenceCapable > > >> @Version(strategy=VersionStrategy.VERSION_NUMBER) > > >> public class MyClass > > >> { > > >> ... > > >> } > > >> > > >> Do you know about how to set it globally? > > >> > > >> > > >> On Tue, 2016-05-31 at 18:54 +0000, César Camilo Lugo Marcos wrote: > > >> > Hello Dan. Answers to your questions: > > >> > > > >> > - which version of Isis are you on? > > >> > 1.12.0 > > >> > - how have you implemented the stress test? is it automated, eg > > >> JMeter, or just adhoc manual testing? > > >> > Using JMeter. We recorded a quite simple script with about 6 > read > > >> only operations using th Wicket viewer. All samplers are the recorded > HTTP. > > >> > - is this stress testing the app running locally on Tomcat, or up in > > >> the cloud (on Amazon Elastic Beanstalk)? > > >> > We have it installed in the AWS Elastic Beanstalk with > PostgreSQL. > > >> > We also made a test on a local host with Tomcat and MySQL, > > >> obtaining very similar results. > > >> > > - how much DB traffic does a single request create? Is it all > > >> read-only; or are there also writes? > > >> > Quite low traffic, About 6 read only operations per user. > > >> > - for reads, are you seeing any N+1 issues? if so, have you tried > to > > >> prevent them using DataNucleus eager loading hint (the "fetch groups" > > >> concept)? > > >> > Not sure what N+1 is. We are letting Datanucleous handle all > > >> foreign keys and primary keys. We have not configured any fetch > groups or > > >> any other configurations than defaults. > > >> > - for writes, are they expected? > > >> > Did not understand this question. Please explain. > > >> > - have you configured any of the various addons that could be > writing > > >> to the DB, eg auditing, command or publishing? are these perhaps > enabled > > >> for safe/query-only actions? If so for any looping within the app > itself, > > >> can you eliminate using the QueryResultsCache? > > >> > We are only using the logging add-on, to log login and logout > > >> operations. No auditing, command or publishing add ons. > > >> > - what data volumes is this on? If large volumes of data, are there > > >> perhaps missing indices? What's the performance like with small > volumes of > > >> data? > > >> > No Data Object or table has more that a few tens of records. > > >> > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? > > >> What's the performance like of one versus the other > > >> > This is PostgreSQL. Also tried MySQL with very similar results. > > >> Have not tried in memory HSQLDB. > > >> > > > >> > > > >> > Cesar. > > >> > > > >> > > > >> > On Tue, 2016-05-31 at 18:53 +0100, Dan Haywood wrote: > > >> > > > >> > > > >> > Hi Cesar, > > >> > > > >> > Those action semantics don't play a large part in the transaction > > >> > management; they are mostly used to determine which type of HTTP > method > > >> to > > >> > expose the action under (GET, PUT or POST) and there is also the > > >> constraint > > >> > that contributed/mixin properties/collections can only be supplied > by > > >> SAFE > > >> > actions. The _ARE_YOU_SURE variants only impact the Wicket > viewer. The > > >> > _REQUEST_CACHEABLE is only honoured if the action is invoked via the > > >> > wrapper factory. > > >> > > > >> > As Martin and Jeroen have said, using JProfiler or similar will > likely > > >> > yield valuable info. > > >> > > > >> > If the issue is with database concurrency (which does seem like a > good > > >> > hypothesis), then for the most part that is under the management of > > >> > DataNucleus itself. There are a bunch of configuration properties > that > > >> can > > >> > be specified in persistor_datanucleus.properties (or > isis.properties, > > >> > actually); we strip off any "isis.persistor.datanucleus.impl" > prefix and > > >> > pass the rest of the key down to DN to configure. > > >> > > > >> > Questions for you: > > >> > - which version of Isis are you on? > > >> > - how have you implemented the stress test? is it automated, eg > > >> JMeter, or > > >> > just adhoc manual testing? > > >> > - is this stress testing the app running locally on Tomcat, or up > in the > > >> > cloud (on Amazon Elastic Beanstalk)? > > >> > - how much DB traffic does a single request create? Is it all > > >> read-only; > > >> > or are there also writes? > > >> > - for reads, are you seeing any N+1 issues? if so, have you tried > to > > >> > prevent them using DataNucleus eager loading hint (the "fetch > groups" > > >> > concept)? > > >> > - for writes, are they expected? > > >> > - have you configured any of the various addons that could be > writing to > > >> > the DB, eg auditing, command or publishing? are these perhaps > enabled > > >> for > > >> > safe/query-only actions? If so > > >> > - for any looping within the app itself, can you eliminate using the > > >> > QueryResultsCache? > > >> > - what data volumes is this on? If large volumes of data, are there > > >> > perhaps missing indices? What's the performance like with small > > >> volumes of > > >> > data? > > >> > - is this for in-memory HSQLDB or out-of-memory DB (eg Postgres)? > > >> What's > > >> > the performance like of one versus the other > > >> > > > >> > I think what I would look for is fast response times for a 1-user > system > > >> > running locally on Tomcat, with an inmemory database, and low data > > >> > volumes. From there, ramp up in different ways, but change one > thing > > >> at a > > >> > time. > > >> > > > >> > HTH > > >> > Dan > > >> > > > >> > > > >> > > > >> > On 31 May 2016 at 15:12, César Camilo Lugo Marcos < > > >> cesar.l...@sisorg.com.mx<mailto:cesar.l...@sisorg.com.mx>> > > >> > wrote: > > >> > > > >> > > Thank you both for the hint. I am looking into JProfiler. Still, > > >> > > anything about the concurrency control options? > > >> > > > > >> > > Cesar. > > >> > > > > >> > > On Tue, 2016-05-31 at 11:13 +0200, Jeroen van der Wal wrote: > > >> > > > I agree with Martin that profiling is the only way to go. To > > >> illustrate: > > >> > > we > > >> > > > recently made some code 8 times faster by a few simple code > changes > > >> on > > >> > > > bottlenecks revealed by JProfiler. And those were in places that > > >> we've > > >> > > > never guessed. > > >> > > > > > >> > > > On 31 May 2016 at 08:39, Martin Grigorov <mgrigo...@apache.org > > >> <mailto:mgrigo...@apache.org>> wrote: > > >> > > > > > >> > > > > Hi, > > >> > > > > > > >> > > > > To find out where is the problem you will have to use a > profiler > > >> like > > >> > > > > JProfiler, Yourkit, JVisualVM, etc. > > >> > > > > Even some thread dumps would help to see what is going on. > > >> > > > > > > >> > > > > Martin Grigorov > > >> > > > > Wicket Training and Consulting > > >> > > > > https://twitter.com/mtgrigorov > > >> > > > > > > >> > > > > On Mon, May 30, 2016 at 9:00 PM, César Camilo Lugo Marcos < > > >> > > > > cesar.l...@sisorg.com.mx<mailto:cesar.l...@sisorg.com.mx>> > wrote: > > >> > > > > > > >> > > > > > Hello, > > >> > > > > > > > >> > > > > > I would like to add to this topic the following: > > >> > > > > > > > >> > > > > > Most of the transactions we are testing in these stress > tests > > >> are not > > >> > > > > > bound in ACTIONS, but just plain reads or default > transactions > > >> using > > >> > > > > > Apache ISIS wicket viewer defaults. I don't see any place > where > > >> I > > >> > > could > > >> > > > > > define semantics for default read or write transactions not > > >> bound > > >> > > into > > >> > > > > > ACTION methods. Is there any place I should look into for > that? > > >> > > > > > > > >> > > > > > Cesar. > > >> > > > > > > > >> > > > > > > > >> > > > > > On Mon, 2016-05-30 at 18:45 +0000, César Camilo Lugo Marcos > > >> wrote: > > >> > > > > > > Hello, > > >> > > > > > > > > >> > > > > > > We have sarted performing some stress tests to our Apache > ISIS > > >> > > > > > > application. We have found this behavior: > > >> > > > > > > > > >> > > > > > > - If we run 1 concurrent user, average response times to > the > > >> main > > >> > > > > object > > >> > > > > > > reads through the wicket viewer are from 1 to 1.5 seconds. > > >> > > > > > > - If we run 2 concurrent users, same transactions go to > > >> average > > >> > > > > response > > >> > > > > > > times up to 16 to 27 seconds. > > >> > > > > > > - If we run 10 concurrent users, the transactions start to > > >> slow > > >> > > down > > >> > > > > > > significantly until the app server freezes and we have to > > >> restart > > >> > > it. > > >> > > > > > > > > >> > > > > > > As you can see, this is a very significant increase in > > >> response > > >> > > time > > >> > > > > for > > >> > > > > > > such a slight change in user load (from 1 user to 2 > users). > > >> So we > > >> > > are > > >> > > > > > > guessing we should look into the concurrency control. > > >> > > > > > > > > >> > > > > > > In the documentation I have found that the only way to > > >> influence > > >> > > the > > >> > > > > way > > >> > > > > > > Apache ISIS manages transactions and concurrency checking > is > > >> by > > >> > > using > > >> > > > > > > the semantics configuration of the ACTION annotation. > > >> > > > > > > > > >> > > > > > > semantics=SAFE_AND_REQUEST_CACHEABLE > > >> > > > > > > semantics=SAFE > > >> > > > > > > semantics=IDEMPOTENT > > >> > > > > > > semantics=IDEMPOTENT_ARE_YOU_SURE > > >> > > > > > > semantics=NON_IDEMPOTENT > > >> > > > > > > semantics=NON_IDEMPOTENT_ARE_YOU_SURE > > >> > > > > > > > > >> > > > > > > I just wanted to confirm if this is the place to look > into, > > >> or are > > >> > > > > there > > >> > > > > > > any other places where we should be looking into too, to > > >> solve the > > >> > > > > > > performance issue. > > >> > > > > > > > > >> > > > > > > Cesar. > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > >> > > > > >> > > > >> > > > >> > > >> > > > > >
