On 27/10/2020 09:16, Mark Thomas wrote: > On 27/10/2020 04:43, Eric Robinson wrote: > > <snip/> > >>>>>> Any changes in the Nginx configuration in the relevant timescale? >>>>>> >>>> >>>> The last change to the nginx config files was on 8/21. The first >>>> report of problems from the users in question was on 9/16. There is >>>> another set of users on a different tomcat instance who reported >>>> issues around 8/26, 5 days after nginx config change. It seems >>>> unlikely to be related. Also, I can't imagine what nginx could be >>>> sending that would induce the upstream tomcat to behave this way. > > If there is some sort of retaining references to request/response/etc. > at the root of this then that sort of issue is very sensitive to timing. > For example, I have had reliable reproduction cases in the past that > stopped working with the addition of a single debug statement. Any sort > of configuration change might have changed the timing sufficiently to > trigger the issue > > At this point, I'd say the Nginx config change might be a potential > trigger if the root cause is retaining references. > >>>>>> Any updates to the application in the relevant timescale? >>>>>> >>>> >>>> Their application was patched to a newer version on 6/5. > > That seems far enough away to be unlikely. > >>>>>> Any features users started using that hadn't been used before in >>>>>> that timescale? >>>> >>>> That one I couldn't answer, as we are only the hosting facility and we >>>> are not in the loop when it comes to the users' workflow, but it seems >>>> unlikely given the nature of their business. > > Fair enough. That one was a bit of a shot in the dark. > > <snip/> > >>>> 1. Now that you have provided this patch, should I still enable >>>> RECYCLE_FACADES=true? > > I'd recommend yes. At least until the issue is resolved. > >>>> 2. The servers in question are multi-tenanted. There are 17 instances >>>> of tomcat, each running on a different set of ports and out of a >>>> separate directory tree, but they are all started with essentially the >>>> same init script, which exports certain site-specific path variables >>>> and executes tomcat from the specified folder structure. Can you think >>>> of any potential issues where making this change for one instance >>>> could have a negative effect on any of the other instances? Probably >>>> not, but just being careful. I will have these changes implemented >>>> during our nightly maintenance window will begin to gather relevant logs >>> first thing tomorrow! > > I can't think of any side effects. > >>>> >>>> --Eric >>> >>> Mark, the changes have been made per your instructions and tomcat has >>> been restarted. The debug0.log, and debug0.log.lck files were created in the >>> directory, but they currently both have 0 bytes. > > Drat. That suggests something isn't quite right as the logs should start > filling up as soon as a single request is made. I'll double check my > instructions if you could double check your end.
I've clarified a few things in the instructions and confirmed they work with my local 7.0.72 build. Note: You will need to be using the BIO connector Mark > > Konstantin noted there was no source provided. I've pushed the branch to > https://github.com/markt-asf/tomcat/tree/debug-7.0.72 so you can see the > changes I made. > >> Also, RECYCLE_FACADES has been enabled and I confirmed that it is referenced >> in the logs as follows... >> >> INFO: Command line argument: >> -Dorg.apache.catalina.connector.RECYCLE_FACADES=true > > Great. > > Mark > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
