> On 27/10/2020 15:22, Eric Robinson wrote: > >> On 27/10/2020 09:16, Mark Thomas wrote: > >>> On 27/10/2020 04:43, Eric Robinson wrote: > >>> > >>> <snip/> > >>> > >>>>>>>> Any changes in the Nginx configuration in the relevant timescale? > >>>>>>>> > >>>>>> > >>>>>> The last change to the nginx config files was on 8/21. The first > >>>>>> report of problems from the users in question was on 9/16. There > >>>>>> is another set of users on a different tomcat instance who > >>>>>> reported issues around 8/26, 5 days after nginx config change. It > >>>>>> seems unlikely to be related. Also, I can't imagine what nginx > >>>>>> could be sending that would induce the upstream tomcat to behave > this way. > >>> > >>> If there is some sort of retaining references to request/response/etc. > >>> at the root of this then that sort of issue is very sensitive to timing. > >>> For example, I have had reliable reproduction cases in the past that > >>> stopped working with the addition of a single debug statement. Any > >>> sort of configuration change might have changed the timing > >>> sufficiently to trigger the issue > >>> > >>> At this point, I'd say the Nginx config change might be a potential > >>> trigger if the root cause is retaining references. > >>> > >>>>>>>> Any updates to the application in the relevant timescale? > >>>>>>>> > >>>>>> > >>>>>> Their application was patched to a newer version on 6/5. > >>> > >>> That seems far enough away to be unlikely. > >>> > >>>>>>>> Any features users started using that hadn't been used before > >>>>>>>> in that timescale? > >>>>>> > >>>>>> That one I couldn't answer, as we are only the hosting facility > >>>>>> and we are not in the loop when it comes to the users' workflow, > >>>>>> but it seems unlikely given the nature of their business. > >>> > >>> Fair enough. That one was a bit of a shot in the dark. > >>> > >>> <snip/> > >>> > >>>>>> 1. Now that you have provided this patch, should I still enable > >>>>>> RECYCLE_FACADES=true? > >>> > >>> I'd recommend yes. At least until the issue is resolved. > >>> > >>>>>> 2. The servers in question are multi-tenanted. There are 17 > >>>>>> instances of tomcat, each running on a different set of ports and > >>>>>> out of a separate directory tree, but they are all started with > >>>>>> essentially the same init script, which exports certain > >>>>>> site-specific path variables and executes tomcat from the > >>>>>> specified folder structure. Can you think of any potential issues > >>>>>> where making this change for one instance could have a negative > >>>>>> effect on any of the other instances? Probably not, but just > >>>>>> being careful. I will have these changes implemented during our > >>>>>> nightly maintenance window will begin to gather relevant logs > >>>>> first thing tomorrow! > >>> > >>> I can't think of any side effects. > >>> > >>>>>> > >>>>>> --Eric > >>>>> > >>>>> Mark, the changes have been made per your instructions and tomcat > >>>>> has been restarted. The debug0.log, and debug0.log.lck files were > >>>>> created in the directory, but they currently both have 0 bytes. > >>> > >>> Drat. That suggests something isn't quite right as the logs should > >>> start filling up as soon as a single request is made. I'll double > >>> check my instructions if you could double check your end. > >> > >> I've clarified a few things in the instructions and confirmed they > >> work with my local 7.0.72 build. > >> > >> Note: You will need to be using the BIO connector > >> > > > > I had switched to the NIO connector last week. Is that why the logs are > > still > at 0 bytes? > > Yes. I only added the debug logging to BIO. > > Somewhere in a previous thread I recommended switching back to BIO as > the code is simpler and therefore easier to debug. > > Note that my previous analysis of the access log valve entries was based on > the assumption you had switched back to BIO. > > Given everything else is in place, I'd suggest switching back to BIO when you > can, waiting for the issue to re-appear and then looking at the debug logs. > > Mark > >
Oh man, I must have missed the recommendation to switch back to BIO. I will do that ASAP, most likely this evening. My apologies for causing the wasted effort. > > > > > >> Mark > >> > >>> > >>> Konstantin noted there was no source provided. I've pushed the > >>> branch to > >>> https://github.com/markt-asf/tomcat/tree/debug-7.0.72 so you can see > >>> the changes I made. > >>> > >>>> Also, RECYCLE_FACADES has been enabled and I confirmed that it is > >> referenced in the logs as follows... > >>>> > >>>> INFO: Command line argument: > >>>> -Dorg.apache.catalina.connector.RECYCLE_FACADES=true > >>> > >>> Great. > >>> > >>> Mark > >>> > >>> -------------------------------------------------------------------- > >>> - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > > Disclaimer : This email and any files transmitted with it are confidential > > and > intended solely for intended recipients. If you are not the named addressee > you should not disseminate, distribute, copy or alter this email. Any views or > opinions presented in this email are solely those of the author and might not > represent those of Physician Select Management. Warning: Although > Physician Select Management has taken reasonable precautions to ensure > no viruses are present in this email, the company cannot accept responsibility > for any loss or damage arising from the use of this email or attachments. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
