Good ideas for blogs.apache.org. My comments are below.
On Tue, Jul 15, 2014 at 4:45 AM, Rainer Jung <rainer.j...@kippdata.de> wrote: > I suggest the following changes to our roller installation in order to > hunt down the instability problems we observe. I'm ready to do the changes > and investigations myself but wanted to first check, whether there are any > objections. > > 1) Trivial: Add response duration to Tomcat access log > > Add "%D" at end of pattern string in server.xml access log configuration. > > 2) Medium: Add GC log > > To rule out GC problems (we don't expect them, but I would like to make > sure), add a couple of startup parameters that write sensible GC data to a > log. > > Note that GC logs have no nice way of rotation. Even the new JVM flags > allowing GC log rotation will overwrite GC log files when the process > starts. So the theoretical risk is writing big log files, but since roller > is restarted relatively frequently, getting rid of the log by removing the > new startup parameters is easy. > > 3) Medium: Add analysis code > > I'd like to deploy an additional app inside the roller Tomcat, that tracks > long running requests and writes our stack information only for the threads > that have long running requests on them. The app is not open source but my > own IP, so no licensing issue here. The protocol is written as a rotating > log file, that can then be investigated. > > The app is packaged as a normal webapp (war) but it doesn't actually > provide a useful web interface. Packaging as a war is just an easy way of > getting it deployed inside Tomcat. > > I have that code being running in important and high-load applications > since a few years, so i see no risk of negatively ifluencinf stability or > performance. It is not doing a tracing, it only collects snapshots of > threads doing long running since in regular time intervals like once per > minute. > > 4) Trivial: Turn off the https connector > > The Tomcat https connector is currently configured only partially and > complains on startup. We don't currently use it, because the ssl endpoint > is on erebus-ssl which forward to the normal http port. > > 5) Medium: Enabling Tomcat Manager Webapp with role manager-jmx > > Enabling Tomcat Manager webapp with a user in role manager-jmx with a > hashed password in tomcat-users.xml. > > This gives me the opportunity to collect JMX information on the fly via > simple lightweight scripts. > > 6) More complex: Caching > > Since roller has only small content, caching would be technically cheap. > Unfortunately currently the response cache headers are not nice to help > caching. So if we want to go down this road, we would have to configura or > fix roller to set useful caching headers (or overwrite them by Tomcat). > Plus we would need to setup the cache, being it inside the Apache reverse > proxy in front of roller, or a separate component. > > I'd like to execute quickly on 1-5 and investigate the collected data. > > Note that recently the db connection pooling was again turned off, because > problems were observed. > These all sound like good ideas and I'd be happy to help with preparing builds, deploying changes, etc. Also, there is a new Roller 5.1 release on the way that fixes the bug that prevented us from using database connection pooling. For caching, note Roller has built-in support for caching for pages and feeds. The cache sizes can be adjusted and it's also possible to plugin memcached as the caching mechanism. Thanks, - Dave
