tomcat 8.5 TLS cipher strings
We're running Tomcat 8.5, currently configured with the following OpenSSL cipher strings in our SSLHostConfig: ciphers="HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!kRSA:!PSK" However, SSLLabs' server test reports that the following available ciphers are weak: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 TLS_DHE_RSA_WITH_AES_128_CBC_SHA Is there a cipher string that concisely includes these *_RSA_WITH_AES_*_CBC_SHA* ciphers that may be used to disallow them in the SSLHostConfig ciphers parameter? I didn't see one in the OpenSSL ciphers reference. SSLLabs reports only the following available ciphers as non-weak: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 Our ciphers are provided via Oracle's JDK 8. -- UH Information Technology Services : Identity & Access Mgmt, Middleware minutas cantorum, minutas balorum, minutas carboratum desendus pantorum - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Weirdest Tomcat Behavior Ever?
Eric sent me a copy of the strace (thanks Eric) and while it is consistent with what has already been observed, it didn't provide any new information on the socket / file descriptor being closed. I'd like to suggest running again with the following: sudo strace -r -f -e trace=network,desc -p That should log the file descriptor being closed (and other fd activity). There are a couple of things we might be able to do with this: - we'll be able to determine if the socket is closed on the same or a different thread - we might be able to correlate the time of closure with other logs (seems unlikely as we have this from Wireshark but you never know) - the class before the close might be enlightening Mark On 13/11/2020 22:05, Paul Carter-Brown wrote: > lol, and there I was feeling ignored :-) > > That was the first thing I would have looked at. Is the OS reporting errors > to the JVM writing data or is the JVM not writing the data. Strace will > tell you this quite easily. > > > On Fri, Nov 13, 2020 at 5:27 PM Eric Robinson > wrote: > >> >>> -Original Message- >>> From: Paul Carter-Brown >>> Sent: Friday, October 16, 2020 6:11 AM >>> To: Tomcat Users List >>> Subject: Re: Weirdest Tomcat Behavior Ever? >>> >>> Hi Eric, >>> >>> These weird situations are sometimes best looked at by confirming what >> the >>> OS is seeing from user-space. >>> >>> Can you run: sudo strace -r -f -e trace=network -p >>> >>> You can then log that to a file and correlate and see if the kernel is >> in fact >>> being asked to send the response. >>> >>> It's very insightful to see what is actually going on between the JVM >> and >>> Kernel. >>> >>> Paul >> >> Paul, this message went to spam and I just found it! >> >> I will try this suggestion immediately. >> >> -Eric >> >>> >>> On Fri, Oct 16, 2020 at 12:16 PM Mark Thomas wrote: >>> On 16/10/2020 10:05, Eric Robinson wrote: > Hi Mark -- > > Those are great questions. See answers below. > > >> -Original Message- >> From: Mark Thomas >> Sent: Friday, October 16, 2020 2:20 AM >> To: users@tomcat.apache.org >> Subject: Re: Weirdest Tomcat Behavior Ever? >> >> On 16/10/2020 00:27, Eric Robinson wrote: >> >> >> >>> The localhost_access log shows a request received and an HTTP 200 >> response sent, as follows... >>> >>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET >>> /app/code.jsp?gizmoid=64438&weekday=5&aptdate=2020-10- >> 15&multiResFacFi >>> >> >>> lterId=0&sessionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztapppr >> oc >> e >>> ssid=40696&rnd2=0.0715816×tamp=15102020125245.789063 >>> HTTP/1.0 >>> ?gizmoid=64438&weekday=5&aptdate=2020-10- >> 15&multiResFacFilterId=0&sess >>> >> >>> ionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappprocessid=40696 >> & >> rn >>> d2=0.0715816×tamp=15102020125245.789063 200 >>> >>> But WireShark shows what really happened. The server received the >>> GET >> request, and then it sent a FIN to terminate the connection. So if tomcat sent >> an HTTP response, it did not make it out the Ethernet card. >>> >>> Is this the weirdest thing or what? Ideas would sure be >> appreciated! >> >> I am assuming there is a typo in your Java version and you are >> using Java 8. >> > > Yes, Java 8. > >> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in >> less than 6 >> months). If you aren't already planning to upgrade (I'd suggest to 9.0.x) then >> you might want to start thinking about it. >> > > Vendor constraint. It's a canned application published by a national software company, and they have not officially approved tomcat 8 for use on Linux yet. > >> I have a few ideas about what might be going on but rather than >> fire out random theories I have some questions that might help >> narrow things down. >> >> 1. If this request was successful, how big is the response? >> > > 1035 bytes. > >> 2. If this request was successful, how long would it typically take >> to complete? >> > > Under 60 ms. > >> 3. Looking at the Wireshark trace for a failed request, how long >> after the last >> byte of the request is sent by the client does Tomcat send the FIN? >> > > Maybe 100 microseconds. > >> 4. Looking at the Wireshark trace for a failed request, is the >> request fully sent >> (including terminating CRLF etc)? >> > > Yes, the request as seen by the tomcat server is complete and is terminated by 0D 0A. > >> 5. Are there any proxies, firewalls etc between the user agent and Tomcat? >> > > User agent -> firewall -> nginx plus -> upstream tomcat servers > >> 6. What timeouts are configured for the Connector?
Re: NGINX + tomcat 8.0.35 (110: Connection timed out)
Chris, I have changed the connector config as below, it has improved the performance. I want to use this config to support at least 20k concurrent requests. I have tested this config and there is a delay in the response and noticed that it's coming from elastic search. I am trying to increase the number of replicas for elastic search to improve the performance. Could you please verify if the below connector config is good enough if I exclude elastic search tuning ? On Thu, Nov 12, 2020 at 8:12 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > Ayub, > > On 11/12/20 11:20, Ayub Khan wrote: > > Chris, > > > > That's correct, it's just a plain static hello world page I created to > > verify tomcat. It is served by tomcat. I have bundled this page in the > same > > context where the service is running. When I create load on the service > and > > then try to access the static hello world page browser keeps busy and > does > > not return the page. > > > > I checked the database dashboard and the monitoring charts are normal, no > > spikes on cpu or any other resources of the database. The delay is > > noticeable when there are more than 1000 concurrent requests from each > of 4 > > different JMeter test instances > > That's 4000 concurrent requests. Your only has 2000 threads, > so only 2000 requests can be processed simultaneously. > > You have a keepalive timeout of 6 seconds (6000ms) and I'm guessing your > load test doesn't actually use KeepAlive. > > > Why does tomcat not even serve the html page > > I think the keepalive timeout explains what you are seeing. > > Are you instructing JMeter to re-use connections and also use KeepAlive? > > What happens if you set the KeepAlive timeout to 1 second instead of 6? > Does that improve things? > > -chris > > > On Thu, Nov 12, 2020 at 7:01 PM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > >> Ayub, > >> > >> On 11/12/20 10:47, Ayub Khan wrote: > >>> Chris, > >>> > >>> I am using hikaricp connection pooling and the maximum pool size is set > >> to > >>> 100, without specifying minimum idle connections. Even during high > load I > >>> see there are more than 80 connections in idle state. > >>> > >>> I have setup debug statements to print the total time taken to complete > >> the > >>> request. The response time of completed call during load is around 5 > >>> seconds, the response time without load is around 400 to 500 > milliseconds > >> > >> That's a significant difference. Is your database server showing high > >> CPU usage or more I/O usage during those high-load times? > >> > >>> During the load I cannot even access static html page > >> > >> Now *that* is an interesting data point. > >> > >> You are sure that the "static" request doesn't hit any other resources? > >> No filter is doing anything? No logging to an external service or > >> double-checking any security constraints in the db before serving the > page? > >> > >> (And the static page is being returned by Tomcat, not nginx, right?) > >> > >> -chris > >> > >>> On Thu, Nov 12, 2020 at 4:59 PM Christopher Schultz < > >>> ch...@christopherschultz.net> wrote: > >>> > Ayub, > > On 11/11/20 16:16, Ayub Khan wrote: > > I was load testing using the ec2 load balancer dns. I have increased > >> the > > connector timeout to 6000 and also gave 32gig to the JVM of tomcat. I > >> am > > not seeing connection timeout in nginx logs now. No errors in > >> kernel.log > I > > am not seeing any errors in tomcat catalina.out. > > The timeouts are most likely related to the connection timeout (and > therefore keepalive) setting. If you are proxying connections from > nginx > and they should be staying open, you should really never be > experiencing > a timeout between nginx and Tomcat. > > > During regular operations when the request count is between 4 to 6k > > requests per minute the open files count for the tomcat process is > between > > 200 to 350. Responses from tomcat are within 5 seconds. > > Good. > > > If the requests count goes beyond 6.5 k open files slowly move up to > 2300 > > to 3000 and the request responses from tomcat become slow. > > This is pretty important, here. You are measuring two things: > > 1. Rise in file descriptor count > 2. Application slowness > > You are assuming that #1 is causing #2. It's entirely possible that #2 > is causing #1. > > The real question is "why is the application slowing down". Do you see > CPU spikes? If not, check your db connections. > > If your db connection pool is fully-utilized (no more available), then > you may have lots of request processing threads sitting there waiting > on > db connections. You'd see a rise in incoming connections (waiting) > which > aren't making any progress, and the application seems to "slow down", > and ther
Re: Weirdest Tomcat Behavior Ever?
lol, and there I was feeling ignored :-) That was the first thing I would have looked at. Is the OS reporting errors to the JVM writing data or is the JVM not writing the data. Strace will tell you this quite easily. On Fri, Nov 13, 2020 at 5:27 PM Eric Robinson wrote: > > > -Original Message- > > From: Paul Carter-Brown > > Sent: Friday, October 16, 2020 6:11 AM > > To: Tomcat Users List > > Subject: Re: Weirdest Tomcat Behavior Ever? > > > > Hi Eric, > > > > These weird situations are sometimes best looked at by confirming what > the > > OS is seeing from user-space. > > > > Can you run: sudo strace -r -f -e trace=network -p > > > > You can then log that to a file and correlate and see if the kernel is > in fact > > being asked to send the response. > > > > It's very insightful to see what is actually going on between the JVM > and > > Kernel. > > > > Paul > > Paul, this message went to spam and I just found it! > > I will try this suggestion immediately. > > -Eric > > > > > On Fri, Oct 16, 2020 at 12:16 PM Mark Thomas wrote: > > > > > On 16/10/2020 10:05, Eric Robinson wrote: > > > > Hi Mark -- > > > > > > > > Those are great questions. See answers below. > > > > > > > > > > > >> -Original Message- > > > >> From: Mark Thomas > > > >> Sent: Friday, October 16, 2020 2:20 AM > > > >> To: users@tomcat.apache.org > > > >> Subject: Re: Weirdest Tomcat Behavior Ever? > > > >> > > > >> On 16/10/2020 00:27, Eric Robinson wrote: > > > >> > > > >> > > > >> > > > >>> The localhost_access log shows a request received and an HTTP 200 > > > >> response sent, as follows... > > > >>> > > > >>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET > > > >>> /app/code.jsp?gizmoid=64438&weekday=5&aptdate=2020-10- > > > >> 15&multiResFacFi > > > >>> > > > >> > > lterId=0&sessionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztapppr > > > >> oc > > > >> e > > > >>> ssid=40696&rnd2=0.0715816×tamp=15102020125245.789063 > > HTTP/1.0 > > > >>> ?gizmoid=64438&weekday=5&aptdate=2020-10- > > > >> 15&multiResFacFilterId=0&sess > > > >>> > > > >> > > ionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappprocessid=40696 > > > >> & > > > >> rn > > > >>> d2=0.0715816×tamp=15102020125245.789063 200 > > > >>> > > > >>> But WireShark shows what really happened. The server received the > > > >>> GET > > > >> request, and then it sent a FIN to terminate the connection. So if > > > tomcat sent > > > >> an HTTP response, it did not make it out the Ethernet card. > > > >>> > > > >>> Is this the weirdest thing or what? Ideas would sure be > appreciated! > > > >> > > > >> I am assuming there is a typo in your Java version and you are > > > >> using > > > Java 8. > > > >> > > > > > > > > Yes, Java 8. > > > > > > > >> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in > > > >> less > > > than 6 > > > >> months). If you aren't already planning to upgrade (I'd suggest to > > > 9.0.x) then > > > >> you might want to start thinking about it. > > > >> > > > > > > > > Vendor constraint. It's a canned application published by a national > > > software company, and they have not officially approved tomcat 8 for > > > use on Linux yet. > > > > > > > >> I have a few ideas about what might be going on but rather than > > > >> fire out random theories I have some questions that might help > > > >> narrow things > > > down. > > > >> > > > >> 1. If this request was successful, how big is the response? > > > >> > > > > > > > > 1035 bytes. > > > > > > > >> 2. If this request was successful, how long would it typically take > > > >> to complete? > > > >> > > > > > > > > Under 60 ms. > > > > > > > >> 3. Looking at the Wireshark trace for a failed request, how long > > > >> after > > > the last > > > >> byte of the request is sent by the client does Tomcat send the FIN? > > > >> > > > > > > > > Maybe 100 microseconds. > > > > > > > >> 4. Looking at the Wireshark trace for a failed request, is the > > > >> request > > > fully sent > > > >> (including terminating CRLF etc)? > > > >> > > > > > > > > Yes, the request as seen by the tomcat server is complete and is > > > terminated by 0D 0A. > > > > > > > >> 5. Are there any proxies, firewalls etc between the user agent and > > > Tomcat? > > > >> > > > > > > > > User agent -> firewall -> nginx plus -> upstream tomcat servers > > > > > > > >> 6. What timeouts are configured for the Connector? > > > >> > > > > > > > > Sorry, which connector are you referring to? > > > > > > > >> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? > > > >> > > > > > > > > HTTP/1.1 > > > > > > > >> 8. Where are you running Wireshark? User agent? Tomcat? Somewhere > > > >> else? > > > > > > > > On the nginx proxy and both upstream tomcat servers. (On the user > > > > agent, > > > too, but that doesn't help us in this case.) > > > > > > > > If you would like to see a screen shot showing all 4 captures > > > side-by-size, I can send you a secure link. It will verify my answers > > > above. It shows 4 separate
RE: Weirdest Tomcat Behavior Ever?
> From: Thomas Meyer > Sent: Friday, November 13, 2020 9:37 AM > To: Tomcat Users List ; Mark Thomas > ; users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > > > Am 13. November 2020 10:06:18 MEZ schrieb Mark Thomas > : > >On 12/11/2020 14:19, Eric Robinson wrote: > >>> From: Mark Thomas > > > > > > > >>> I keep coming back to this. Something triggered this problem (note > >that > >>> trigger not necessarily the same as root cause). Given that the app, > >Tomcat > >>> and JVM versions didn't change that again points to some other > >component. > >>> > >> > >> Perfectly understandable. It's the oldest question in the diagnostic > >playbook. What changed? I wish I had an answer. Whatever it was, if > >impacted both upstream servers. > >> > >>> Picking just one of the wild ideas I've had is there some sort of > >firewall, IDS, > >>> IPS etc. that might be doing connection tracking and is, for some > >reason, > >>> getting it wrong and closing the connection in error? > >>> > >> > >> Three is no firewall or IDS software running on the upstreams. The > >only thing that comes to mind that may have been installed during that > >timeframe is Sophos antivirus and Solar Winds RMM. Sophos was the first > >thing I disabled when I saw the packet issues. > > > >ACK. > > > > The aim with this logging is to provide evidence of whether or not > > there is a file descriptor handling problem in the JRE. My > > expectation is that with these logs we will have reached the limit > >of > > what we can do with Tomcat but will be able to point you in the > >right > >>> direction for further investigation. > > > >I've had a chance to review these logs. > > > >To answer your (offlist) question about the HTTP/1.1 vs. HTTP/1.0 in > >the Nginx logs I *think* the Nginx logs are showing that the request > >received by Nginx is using HTTP/1.1. > > > >The logging does not indicate any issue with Java's handling of file > >descriptors. The file descriptor associated with the socket where the > >request fails is only observed to be associated with the socket where > >the request fails. There is no indication that the file descriptor is > >corrupted nor is there any indication that another thread tries to use > >the same file descriptor. > > > >I dug a little into the exception where the write fails: > > > >java.net.SocketException: Bad file descriptor (Write failed) > >at java.net.SocketOutputStream.socketWrite0(Native Method) > >at > >java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) > >at > java.net.SocketOutputStream.write(SocketOutputStream.java:155) > >at > >org.apache.tomcat.util.net.JIoEndpoint$DebugOutputStream.write(JIoEnd > point.java:1491) > >at > >org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOu > tputBuffer.java:247) > >at > >org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) > >at > >org.apache.coyote.http11.InternalOutputBuffer.endRequest(InternalOutp > ut > >Buffer.java:183) > >... > > > > > >I took a look at the JRE source code. That exception is triggered by an > >OS level error (9, EBADF, "Bad file descriptor") when the JRE makes the > >OS call to write to the socket. > > > >Everything I have found online points to one of two causes for such an > >error: > >a) the socket has already been closed > >b) the OS has run out of file descriptors > > Was it mentioned what OS is used? What Linux kernel version? > Are any security modules like SELinux or similar is in use? > It's maybe possible that a tracepoint exists that can be activated to get > better > understanding when the OS closes the socket. > Way back at the beginning of the thread. 😊 CentOS Linux release 7.8.2003 (Core) [root@001app01a ~]# uname -a Linux 001app01a.ccnva.local 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@001app01a ~]# sestatus SELinux status: enabled SELinuxfs mount:/sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: permissive Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 31 > > > >There is no indication that the JRE or Tomcat or the application is > >doing a) Previous investigations have ruled out b) > > > >The wireshark trace indicates that the socket is closed before the > >write takes place which suggests a) rather more than b). Even so, I'd > >be tempted to double check b) and maybe try running Tomcat with > >-XX:+MaxFDLimit just to be sure. > > > >If you haven't already, I think now is the time to follow Paul > >Carter-Brown's advice from earlier in this thread and use strace to see > >what is going on between the JRE and the OS. The aim being to answer > >the question "what is triggering the socket close" > > > >I can try and help interpret that log but I am far from
Re: Weirdest Tomcat Behavior Ever?
Am 13. November 2020 10:06:18 MEZ schrieb Mark Thomas : >On 12/11/2020 14:19, Eric Robinson wrote: >>> From: Mark Thomas > > > >>> I keep coming back to this. Something triggered this problem (note >that >>> trigger not necessarily the same as root cause). Given that the app, >Tomcat >>> and JVM versions didn't change that again points to some other >component. >>> >> >> Perfectly understandable. It's the oldest question in the diagnostic >playbook. What changed? I wish I had an answer. Whatever it was, if >impacted both upstream servers. >> >>> Picking just one of the wild ideas I've had is there some sort of >firewall, IDS, >>> IPS etc. that might be doing connection tracking and is, for some >reason, >>> getting it wrong and closing the connection in error? >>> >> >> Three is no firewall or IDS software running on the upstreams. The >only thing that comes to mind that may have been installed during that >timeframe is Sophos antivirus and Solar Winds RMM. Sophos was the first >thing I disabled when I saw the packet issues. > >ACK. > > The aim with this logging is to provide evidence of whether or not > there is a file descriptor handling problem in the JRE. My > expectation is that with these logs we will have reached the limit >of > what we can do with Tomcat but will be able to point you in the >right >>> direction for further investigation. > >I've had a chance to review these logs. > >To answer your (offlist) question about the HTTP/1.1 vs. HTTP/1.0 in >the >Nginx logs I *think* the Nginx logs are showing that the request >received by Nginx is using HTTP/1.1. > >The logging does not indicate any issue with Java's handling of file >descriptors. The file descriptor associated with the socket where the >request fails is only observed to be associated with the socket where >the request fails. There is no indication that the file descriptor is >corrupted nor is there any indication that another thread tries to use >the same file descriptor. > >I dug a little into the exception where the write fails: > >java.net.SocketException: Bad file descriptor (Write failed) > at java.net.SocketOutputStream.socketWrite0(Native Method) > at >java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) > at java.net.SocketOutputStream.write(SocketOutputStream.java:155) > at >org.apache.tomcat.util.net.JIoEndpoint$DebugOutputStream.write(JIoEndpoint.java:1491) > at >org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:247) > at >org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) > at >org.apache.coyote.http11.InternalOutputBuffer.endRequest(InternalOutputBuffer.java:183) >... > > >I took a look at the JRE source code. That exception is triggered by an >OS level error (9, EBADF, "Bad file descriptor") when the JRE makes the >OS call to write to the socket. > >Everything I have found online points to one of two causes for such an >error: >a) the socket has already been closed >b) the OS has run out of file descriptors Was it mentioned what OS is used? What Linux kernel version? Are any security modules like SELinux or similar is in use? It's maybe possible that a tracepoint exists that can be activated to get better understanding when the OS closes the socket. > >There is no indication that the JRE or Tomcat or the application is >doing a) >Previous investigations have ruled out b) > >The wireshark trace indicates that the socket is closed before the >write >takes place which suggests a) rather more than b). Even so, I'd be >tempted to double check b) and maybe try running Tomcat with >-XX:+MaxFDLimit just to be sure. > >If you haven't already, I think now is the time to follow Paul >Carter-Brown's advice from earlier in this thread and use strace to see >what is going on between the JRE and the OS. The aim being to answer >the >question "what is triggering the socket close" > >I can try and help interpret that log but I am far from an expert. You >may want to seek help elsewhere. > >Mark > >- >To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >For additional commands, e-mail: users-h...@tomcat.apache.org -- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Weirdest Tomcat Behavior Ever?
> -Original Message- > From: Paul Carter-Brown > Sent: Friday, October 16, 2020 6:11 AM > To: Tomcat Users List > Subject: Re: Weirdest Tomcat Behavior Ever? > > Hi Eric, > > These weird situations are sometimes best looked at by confirming what the > OS is seeing from user-space. > > Can you run: sudo strace -r -f -e trace=network -p > > You can then log that to a file and correlate and see if the kernel is in fact > being asked to send the response. > > It's very insightful to see what is actually going on between the JVM and > Kernel. > > Paul Paul, this message went to spam and I just found it! I will try this suggestion immediately. -Eric > > On Fri, Oct 16, 2020 at 12:16 PM Mark Thomas wrote: > > > On 16/10/2020 10:05, Eric Robinson wrote: > > > Hi Mark -- > > > > > > Those are great questions. See answers below. > > > > > > > > >> -Original Message- > > >> From: Mark Thomas > > >> Sent: Friday, October 16, 2020 2:20 AM > > >> To: users@tomcat.apache.org > > >> Subject: Re: Weirdest Tomcat Behavior Ever? > > >> > > >> On 16/10/2020 00:27, Eric Robinson wrote: > > >> > > >> > > >> > > >>> The localhost_access log shows a request received and an HTTP 200 > > >> response sent, as follows... > > >>> > > >>> 10.51.14.133 [15/Oct/2020:12:52:45 -0400] 57 GET > > >>> /app/code.jsp?gizmoid=64438&weekday=5&aptdate=2020-10- > > >> 15&multiResFacFi > > >>> > > >> > lterId=0&sessionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztapppr > > >> oc > > >> e > > >>> ssid=40696&rnd2=0.0715816×tamp=15102020125245.789063 > HTTP/1.0 > > >>> ?gizmoid=64438&weekday=5&aptdate=2020-10- > > >> 15&multiResFacFilterId=0&sess > > >>> > > >> > ionDID=0&GzUserId=71340&zztusrlogtblid=321072&zztappprocessid=40696 > > >> & > > >> rn > > >>> d2=0.0715816×tamp=15102020125245.789063 200 > > >>> > > >>> But WireShark shows what really happened. The server received the > > >>> GET > > >> request, and then it sent a FIN to terminate the connection. So if > > tomcat sent > > >> an HTTP response, it did not make it out the Ethernet card. > > >>> > > >>> Is this the weirdest thing or what? Ideas would sure be appreciated! > > >> > > >> I am assuming there is a typo in your Java version and you are > > >> using > > Java 8. > > >> > > > > > > Yes, Java 8. > > > > > >> That Tomcat version is over 3.5 years old (and Tomcat 7 is EOL in > > >> less > > than 6 > > >> months). If you aren't already planning to upgrade (I'd suggest to > > 9.0.x) then > > >> you might want to start thinking about it. > > >> > > > > > > Vendor constraint. It's a canned application published by a national > > software company, and they have not officially approved tomcat 8 for > > use on Linux yet. > > > > > >> I have a few ideas about what might be going on but rather than > > >> fire out random theories I have some questions that might help > > >> narrow things > > down. > > >> > > >> 1. If this request was successful, how big is the response? > > >> > > > > > > 1035 bytes. > > > > > >> 2. If this request was successful, how long would it typically take > > >> to complete? > > >> > > > > > > Under 60 ms. > > > > > >> 3. Looking at the Wireshark trace for a failed request, how long > > >> after > > the last > > >> byte of the request is sent by the client does Tomcat send the FIN? > > >> > > > > > > Maybe 100 microseconds. > > > > > >> 4. Looking at the Wireshark trace for a failed request, is the > > >> request > > fully sent > > >> (including terminating CRLF etc)? > > >> > > > > > > Yes, the request as seen by the tomcat server is complete and is > > terminated by 0D 0A. > > > > > >> 5. Are there any proxies, firewalls etc between the user agent and > > Tomcat? > > >> > > > > > > User agent -> firewall -> nginx plus -> upstream tomcat servers > > > > > >> 6. What timeouts are configured for the Connector? > > >> > > > > > > Sorry, which connector are you referring to? > > > > > >> 7. Is this HTTP/1.1, HTTP/2, AJP, with or without TLS? > > >> > > > > > > HTTP/1.1 > > > > > >> 8. Where are you running Wireshark? User agent? Tomcat? Somewhere > > >> else? > > > > > > On the nginx proxy and both upstream tomcat servers. (On the user > > > agent, > > too, but that doesn't help us in this case.) > > > > > > If you would like to see a screen shot showing all 4 captures > > side-by-size, I can send you a secure link. It will verify my answers > > above. It shows 4 separate WireShark captures taken simultaneously: > > > > > > (a) the request going from the nginx proxy to tomcat 1 > > > (b) tomcat 1 receiving the request and terminating the connection > > > (c) nginx sending the request to tomcat 2 > > > (d) tomcat 2 replying to the request (but the reply does not help > > > the > > user because the tomcat server does not recognize the user agent's > > JSESSIONID cookie, so it responds "invalid session." > > > > Hmm. > > > > That rules out most of my ideas. > > > > I'd like to see those screen shots please. Better still would be > > access to the captu
RE: Weirdest Tomcat Behavior Ever?
> -Original Message- > From: Mark Thomas > Sent: Friday, November 13, 2020 3:06 AM > To: users@tomcat.apache.org > Subject: Re: Weirdest Tomcat Behavior Ever? > > On 12/11/2020 14:19, Eric Robinson wrote: > >> From: Mark Thomas > > > > >> I keep coming back to this. Something triggered this problem (note > >> that trigger not necessarily the same as root cause). Given that the > >> app, Tomcat and JVM versions didn't change that again points to some > other component. > >> > > > > Perfectly understandable. It's the oldest question in the diagnostic > playbook. What changed? I wish I had an answer. Whatever it was, if > impacted both upstream servers. > > > >> Picking just one of the wild ideas I've had is there some sort of > >> firewall, IDS, IPS etc. that might be doing connection tracking and > >> is, for some reason, getting it wrong and closing the connection in error? > >> > > > > Three is no firewall or IDS software running on the upstreams. The only > thing that comes to mind that may have been installed during that timeframe > is Sophos antivirus and Solar Winds RMM. Sophos was the first thing I > disabled when I saw the packet issues. > > ACK. > > The aim with this logging is to provide evidence of whether or not > there is a file descriptor handling problem in the JRE. My > expectation is that with these logs we will have reached the limit > of what we can do with Tomcat but will be able to point you in the > right > >> direction for further investigation. > > I've had a chance to review these logs. > > To answer your (offlist) question about the HTTP/1.1 vs. HTTP/1.0 in the > Nginx logs I *think* the Nginx logs are showing that the request received by > Nginx is using HTTP/1.1. > > The logging does not indicate any issue with Java's handling of file > descriptors. The file descriptor associated with the socket where the request > fails is only observed to be associated with the socket where the request > fails. There is no indication that the file descriptor is corrupted nor is > there > any indication that another thread tries to use the same file descriptor. > > I dug a little into the exception where the write fails: > > java.net.SocketException: Bad file descriptor (Write failed) > at java.net.SocketOutputStream.socketWrite0(Native Method) > at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) > at > java.net.SocketOutputStream.write(SocketOutputStream.java:155) > at > org.apache.tomcat.util.net.JIoEndpoint$DebugOutputStream.write(JIoEndp > oint.java:1491) > at > org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOut > putBuffer.java:247) > at > org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) > at > org.apache.coyote.http11.InternalOutputBuffer.endRequest(InternalOutput > Buffer.java:183) > ... > > > I took a look at the JRE source code. That exception is triggered by an OS > level > error (9, EBADF, "Bad file descriptor") when the JRE makes the OS call to > write to the socket. > > Everything I have found online points to one of two causes for such an > error: > a) the socket has already been closed > b) the OS has run out of file descriptors > > There is no indication that the JRE or Tomcat or the application is doing a) > Previous investigations have ruled out b) > > The wireshark trace indicates that the socket is closed before the write takes > place which suggests a) rather more than b). Even so, I'd be tempted to > double check b) and maybe try running Tomcat with -XX:+MaxFDLimit just to > be sure. > > If you haven't already, I think now is the time to follow Paul Carter-Brown's > advice from earlier in this thread and use strace to see what is going on > between the JRE and the OS. The aim being to answer the question "what is > triggering the socket close" > This is the second time you alluded to comments from a someone I haven't seen in the thread . I just checked my spam folder and found that, for some unknown reason, 4 messages in this long thread went to spam. They were from Paul Carter-Brown, Konstantin Kolinko, and Daniel Skiles. They probably thought I ignored them. Now I'll go check out their recommendations. > I can try and help interpret that log but I am far from an expert. You may > want to seek help elsewhere. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no v
Re: getopts of Perl is not working Tomcat 9
On 13/11/2020 13:14, Jonnalagadda, Swathi (External) wrote:
> Hi
>
> We have our web application implemented in Perl cgi which is using below
> function of perl. we are upgrading from Tomcat 8.0.21 to Tomcat 9.0.38.
>
> The below function (getopts) is working fine Tomcat 8.0.21 but is not working
> in Tomcat 9.0.38. We even tried Tomcat 9.0.39 as well and it is not working
> even in 9.0.39.
>
> use Getopt::Std;
>
> getopts('ei:npXP');
>
> Could you please suggest a solution here.
How is the CGI servlet configured?
What URL are you using to call the CGI servlet?
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
getopts of Perl is not working Tomcat 9
Hi
We have our web application implemented in Perl cgi which is using below
function of perl. we are upgrading from Tomcat 8.0.21 to Tomcat 9.0.38.
The below function (getopts) is working fine Tomcat 8.0.21 but is not working
in Tomcat 9.0.38. We even tried Tomcat 9.0.39 as well and it is not working
even in 9.0.39.
use Getopt::Std;
getopts('ei:npXP');
Could you please suggest a solution here.
Thanks
Swathi
Re: Weirdest Tomcat Behavior Ever?
On 12/11/2020 14:19, Eric Robinson wrote: >> From: Mark Thomas >> I keep coming back to this. Something triggered this problem (note that >> trigger not necessarily the same as root cause). Given that the app, Tomcat >> and JVM versions didn't change that again points to some other component. >> > > Perfectly understandable. It's the oldest question in the diagnostic > playbook. What changed? I wish I had an answer. Whatever it was, if impacted > both upstream servers. > >> Picking just one of the wild ideas I've had is there some sort of firewall, >> IDS, >> IPS etc. that might be doing connection tracking and is, for some reason, >> getting it wrong and closing the connection in error? >> > > Three is no firewall or IDS software running on the upstreams. The only thing > that comes to mind that may have been installed during that timeframe is > Sophos antivirus and Solar Winds RMM. Sophos was the first thing I disabled > when I saw the packet issues. ACK. The aim with this logging is to provide evidence of whether or not there is a file descriptor handling problem in the JRE. My expectation is that with these logs we will have reached the limit of what we can do with Tomcat but will be able to point you in the right >> direction for further investigation. I've had a chance to review these logs. To answer your (offlist) question about the HTTP/1.1 vs. HTTP/1.0 in the Nginx logs I *think* the Nginx logs are showing that the request received by Nginx is using HTTP/1.1. The logging does not indicate any issue with Java's handling of file descriptors. The file descriptor associated with the socket where the request fails is only observed to be associated with the socket where the request fails. There is no indication that the file descriptor is corrupted nor is there any indication that another thread tries to use the same file descriptor. I dug a little into the exception where the write fails: java.net.SocketException: Bad file descriptor (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at org.apache.tomcat.util.net.JIoEndpoint$DebugOutputStream.write(JIoEndpoint.java:1491) at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:247) at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480) at org.apache.coyote.http11.InternalOutputBuffer.endRequest(InternalOutputBuffer.java:183) ... I took a look at the JRE source code. That exception is triggered by an OS level error (9, EBADF, "Bad file descriptor") when the JRE makes the OS call to write to the socket. Everything I have found online points to one of two causes for such an error: a) the socket has already been closed b) the OS has run out of file descriptors There is no indication that the JRE or Tomcat or the application is doing a) Previous investigations have ruled out b) The wireshark trace indicates that the socket is closed before the write takes place which suggests a) rather more than b). Even so, I'd be tempted to double check b) and maybe try running Tomcat with -XX:+MaxFDLimit just to be sure. If you haven't already, I think now is the time to follow Paul Carter-Brown's advice from earlier in this thread and use strace to see what is going on between the JRE and the OS. The aim being to answer the question "what is triggering the socket close" I can try and help interpret that log but I am far from an expert. You may want to seek help elsewhere. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
