Re: Excessive CPU w/APR Connectors on tomcat-native 1.1.22
Brief follow up on CPU spike issue. In an attempt to work around the problem via configuration changes, we have swapped out APR connectors with NIO using an equivalent configuration. (The only meaningful changes are SSL configuration directives.) Since swapping out connectors over the weekend, we have not had any CPU spikes. That's long enough for me to consider it a suitable workaround. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Excessive CPU w/APR Connectors on tomcat-native 1.1.22
> Can you confirm whether or not the issue exists with 6.0.26 and 1.1.22? I cannot. We have tried repeatedly to reproduce this problem in a test environment where such experimentation is tolerated, but the problem simply does not manifest using available load testing tools. We attempted to try 7.0.23+1.2.20 in production, but I couldn't get a working configuration for the supported SSL/TLS protocols we require. > How long do the periods of high CPU usage last? Varies quite a bit. Based on data from our enterprise monitoring system, the problem lasts as little as 5 min or less and has lasted as much as 25 min. YourKit data from the incidents that have happened in the past 24h shows similarly wide variation: 2m40s 1m38s 13m11s 8s > How sure are you that the CPU is being burned in the threads that are in > sendbb? I'm certain of nothing, but I can share the evidence. YourKit CPU profiling data shows that request processing threads consume the majority of resources during these periods. No surprise there. However, the only method that is consistently executed during these periods is sendbb() by two or more threads. For the last problem period, which happens to be the shortest (8s), catalina-exec threads accounted for 89% of CPU use. Following are the RUNNABLE threads at 1s sample intervals during that period: Stacks at 09:11:01 AM (uptime 1 day 3:26:47) 1. catalina-exec-45 [RUNNABLE, IN_NATIVE] CPU time: 1:57 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:32 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:02 AM (uptime 1 day 3:26:48) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:21 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:33 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:03 AM (uptime 1 day 3:26:49) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:22 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:34 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:04 AM (uptime 1 day 3:26:50) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:23 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:35 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:05 AM (uptime 1 day 3:26:51) 1. catalina-exec-38 [RUNNABLE, IN_NATIVE] CPU time: 1:28 java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) 2. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:24 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 3. catalina-exec-44 [RUNNABLE] CPU time: 1:43 java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int) 4. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:35 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:06 AM (uptime 1 day 3:26:52) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:25 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:36 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:07 AM (uptime 1 day 3:26:53) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:25 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:37 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 3. catalina-exec-52 [RUNNABLE] CPU time: 0:04 java.lang.Throwable.fillInStackTrace() Stacks at 09:11:08 AM (uptime 1 day 3:26:54) 1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:26 org.apache.tomcat.jni.Socket.sendbb(long, int, int) 2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:38 org.apache.tomcat.jni.Socket.sendbb(long, int, int) Stacks at 09:11:09 AM (uptime 1 day 3:26:55) 1. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:39 org.apache.tomcat.jni.Socket.sendbb(long, int, int) After the last sample, CPU usage drops precipitously. Our application is a Web SSO product that writes relatively small response payloads. It's hard to explain two request processing threads that take several seconds to write small response payloads without citing a bug in the servlet container. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Excessive CPU w/APR Connectors on tomcat-native 1.1.22
We are seeing excessive CPU burn (top > 300% on multicore machine) in multiple versions of Tomcat that use APR connectors exclusively. The problem does not correlate with load. We initially saw it on 6.0.35 and subsequently on 7.0.23 as we attempted to upgrade around the problem. We have determined that the component common to both versions is tomcat-native 1.1.22. (We were not seeing this behavior on our previous component mix of 6.0.26/1.1.20.) In addition to the circumstantial evidence of version changes, we have some JVM data that implicate tomcat-native. We've taken thread dumps and YourKit snapshots (w/CPU sampling enabled) during the problem periods and a consistent pattern appears: at least two connector threads are in org.apache.tomcat.jni.Socket.sendbb(long, int, int) throughout the period of CPU churn. The following instrumented thread dump of active threads is illustrative of this pattern: Stacks at 10:57:51 AM (uptime 5:13:38) catalina-exec-1 [WAITING] CPU time: 2:03 sun.misc.Unsafe.park(boolean, long) java.util.concurrent.locks.LockSupport.parkNanos(Object, long) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) java.util.concurrent.LinkedBlockingQueue.poll(long, TimeUnit) org.apache.tomcat.util.threads.TaskQueue.poll(long, TimeUnit) org.apache.tomcat.util.threads.TaskQueue.poll(long, TimeUnit) java.util.concurrent.ThreadPoolExecutor.getTask() java.util.concurrent.ThreadPoolExecutor$Worker.run() java.lang.Thread.run() catalina-exec-10 [RUNNABLE, IN_NATIVE] CPU time: 2:28 org.apache.tomcat.jni.Socket.sendbb(long, int, int) org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer() org.apache.coyote.http11.InternalAprOutputBuffer.access$100(InternalAprOutputBuffer) org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(ByteChunk, Response) org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(ByteChunk, Response) org.apache.coyote.http11.AbstractOutputBuffer.doWrite(ByteChunk, Response) org.apache.coyote.Response.doWrite(ByteChunk) org.apache.catalina.connector.OutputBuffer.realWriteBytes(byte[], int, int) org.apache.tomcat.util.buf.ByteChunk.append(byte[], int, int) org.apache.catalina.connector.OutputBuffer.writeBytes(byte[], int, int) org.apache.catalina.connector.OutputBuffer.write(byte[], int, int) org.apache.catalina.connector.CoyoteOutputStream.write(byte[], int, int) org.apache.catalina.servlets.DefaultServlet.copy(CacheEntry, InputStream, ServletOutputStream) org.apache.catalina.servlets.DefaultServlet.serveResource(HttpServletRequest, HttpServletResponse, boolean) org.apache.catalina.servlets.DefaultServlet.doGet(HttpServletRequest, HttpServletResponse) javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse) javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) edu.vt.middleware.servlet.filter.RequestDumperFilter.doFilter(ServletRequest, ServletResponse, FilterChain) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) com.github.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ServletRequest, ServletResponse, FilterChain) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) edu.vt.middleware.servlet.filter.CharacterEncodingFilter.doFilter(ServletRequest, ServletResponse, FilterChain) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response) org.apache.catalina.core.StandardContextValve.invoke(Request, Response) org.apache.catalina.authenticator.AuthenticatorBase.invoke(Request, Response) org.apache.catalina.core.StandardHostValve.invoke(Request, Response) org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response) org.apache.catalina.valves.AccessLogValve.invoke(Request, Response) org.apache.catalina.core.StandardEngineValve.invoke(Request, Response) org.apache.catalina.connector.CoyoteAdapter.service(Request, Response) org.apache.coyote.http11.AbstractHttp11Processor.process(SocketWrapper) org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(SocketWrapper, SocketStatus) org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run() java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable) java.util.concurrent.ThreadPoolExecutor$Worker.run() java.lang.Thread.run() catalina-exec-11 [RUNNABLE, IN_NATIVE] CPU time: 10:19 org.apache.tomcat.jni.Socket.sendbb(long, int, int) org.apac
Re: Character set issue
> /can/ the servlet (or one of the filters) > do anything that would cause the value of "name1" to /not/ be a correct Java > "TÜV" string in the servlet ? Yes, absolutely. If this is a posted value and some filter fires that coerces the encoding (e.g. request.getParameter() in the case of POST) of the request, all subsequent filters and the servlet will see the string in the encoding of the first filter. This is why it's important to set the encoding as early in the servlet processing pipeline as possible. For your particular case it's hard to imagine an encoding in practice that would make that string appear incorrectly. Both iso-8859-1 and utf-8 should handle Ü correctly. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Do any of the Tomcat LDAP-type realms support "no password" authentication?
> I was wondering if any of the LDAP-type realms (e.g., JNDIRealm, etc.) > support an authentication mode where no password or credentials are required? It's hard to imagine a valid use case for this -- I hope you know what you're doing. That said, you could use JAASRealm with http://code.google.com/p/vt-middleware/source/browse/vt-ldap/tags/vt-ldap-3.3.4/src/main/java/edu/vt/middleware/ldap/jaas/LdapDnAuthorizationModule.java to accomplish this. I should note that the intention is for LdapDnAuthorizationModule to be combined with another module that actually performs authentication (e.g. bind or compare), but you could abuse it for your use case I believe. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CLOSE_WAIT Connection Issue
> 3] And then from Clinet Side, from where, I execute curl, kill all curl > process. > So, on server all ESTABLISHED becomes, CLOSE_WAIT in netstat. I'd imagine kill -KILL or kill -TERM is preventing proper socket teardown. The server is expecting ACKs from the clients that apparently not being sent. That's certainly expected behavior for the KILL signal. I might expect curl to handle the TERM signal gracefully by tearing down the connections before exiting, but I've never tried. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: problem with loading Bouncy Castle
Crypto providers need to be installed into the JRE extension directory
(e.g. $JRE_HOME/lib/ext) and an entry must be added to
$JRE_HOME/lib/security/java.security to install the provider. For
example:
security.provider.5=org.bouncycastle.jce.provider.BouncyCastleProvider
Note the 5 above; each provider needs a unique number and the ordering
determines which provider is used for a given crypto primitive
obtained by code like the following:
Cipher c1 = Cipher.getInstance("DES");
If BC is before the Sun provider, you'll use the BC DES
implementation. Put it last if in doubt. (Although it's arguably
better than Sun in many cases.)
M
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SSL connect to APR fails - "bad version"
The following works as expected on my config (6.0.26) using the default protocols and cipher suite as in your config: $ openssl s_client -connect eiger:443 -debug -ssl3 CONNECTED(0003) ... Something in your SSL version jumped out at me: OpenSSL 0.9.8e-fips-rhel5 Looks like you're running OpenSSL with the FIPS compliance features enabled, which may impose additional requirements on SSL negotiation. I can't provide any further insight, but hopefully it might point to an area for further investigation. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How to save the log info to log file
> In windows, How to let the tomcat write the catalina log to the log file I was going to suggest setting the CATALINA_OUT environment variable, but it looks like that's only supported in catalina.sh for Unix hosts. If you're starting via startup.bat/catalina.bat, you could edit catalina.bat to redirect STDOUT to a file. See around line 286. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: URL "simplification"
> You can also do it with iptables, converting any port 80 request to 8080. > The user types in http://mywebapp but gets redirected to > http://hostname:8080 +1 What I use: -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8080 M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: TOMCAT-7.0 and SSL issues
> My question is, why my first aproach didn't work ? Likely because you specified the APR connector somewhere in your connector config, yet the SSL instructions you cited for CAS describe the setup for the pure-java connectors. APR is a different (but friendly) beast. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How to Configure Tomcat 7.0 for SSL
> SSLEngine="on" There's no such attribute for this connector. The attribute you want is SSLEnabled="true". I would also recommend the following attributes for security-conscious deployments: SSLProtocol="SSLv3+TLSv1" SSLCipherSuite="!ADH:HIGH:MEDIUM:-SSLv2" If you continue to have trouble, post the output of catalina.out. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SSL configuration / APR / Tomcat 7.0.17
> port="8443" maxThreads="200" > scheme="https" secure="true" SSLEnabled="true" > SSLCertificateFile="/usr/local/tomcat/conf/my.crt" > SSLCertificateKeyFile="/usr/local/tomcat/conf/my.pem" > clientAuth="optional" SSLProtocol="TLSv1"/> > Things to check: - my.pem file must be readable the user under which Tomcat runs - Ensure my.pem is a valid private key file using OpenSSL or similar. I don't believe encrypted private keys are supported in the APR connector, so make sure there's no password-based encryption on the key. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SSL issue
If you've got the container to request and accept a client cert, the connector configuration is likely correct and it's an SSL trust problem. Please post the output of the following command: keytool -list -keystore D:/users/apache-tomcat-7.0.11/keystore/trust.keystore -storePass changeit Also, please post some evidence that the client is sending the cert you think it is. OpenSSL s_client is a good choice if you don't already have a good tool for SSL troubleshooting. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Moving init params from web.xml to context.xml
> I can get with the Dev team and see if they are willing to re-code for the > possibility (or even need to). It's a valuable change that dramatically increases your deployment options; well worth the effort IMO. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Moving init params from web.xml to context.xml
> This is equivalent to the inclusion of the following element in the web > application deployment descriptor (/WEB-INF/web.xml): > > > companyName > My Company, Incorporated > > > but does not require modification of the deployment descriptor to customize > this value. That's correct but there's no way afaik to map a context parameter onto a servlet init parameter like these mentioned previously: > > ... > > SERVA_PROPS > /WEB-INF/SERVA.properties > > In my experience the servlet needs to search both context parameters and init parameters for initialization. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: html meta http-equiv content-type not work ?
> I think you mean "any attempt to WRITE The RESPONSE".
I took some time to verify the effect I described,
https://github.com/serac/charset-test. Reading data from the request
body coerces the encoding as I claimed. Simply swap the order of the
filter-mappings in web.xml and post some unicode data to see for
yourself. If ConsumeRequestFilter appears before
CharacterEncodingFilter, the posted data will not be treated as UTF-8.
> Spring's filter only sets the /request/ encoding, not the response.
Looks like it sets response encoding when forceEncoding == true:
if (this.encoding != null && (this.forceEncoding ||
request.getCharacterEncoding() == null)) {
request.setCharacterEncoding(this.encoding);
if (this.forceEncoding) {
response.setCharacterEncoding(this.encoding);
}
}
(3.0.5 source)
M
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: APR connector pollTime defaults are strange in tomcat6/7
> Does Tomcat APR really needs pollTime set so low by default? Anyone care to comment on this point? I'm interested in this discussion as a user of Linux+APR connectors. While we don't yet run on a tickless kernel, I'm considering trying to measure the impact on our systems as well, but some insight on the rationale for the defaults would be helpful. M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: html meta http-equiv content-type not work ?
> If you want to set > the charset in the HTTP header, you'll need to set it explicitly using a > JSP, a filter or similar. It's wise to do both, http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q8. For the filter, it's important to put the filter at or near the top of the filter chain since any attempt to read the request will coerce the encoding, possibly before it is explicitly set. Two good existing filters for this purpose: http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/filter/CharacterEncodingFilter.html http://code.google.com/p/vt-middleware/wiki/vtservletfilters#CharacterEncodingFilter M - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SSL Certificate formats, requirements for import into existing keystore
> There is some "junk" ("bag attributes")n the file that I don't' understand. I
> am used to just seeing "-BEGIN CERTIFICATE- "END CERTIFICATE-
> "" ""-BEGIN RSA PRIVATE KEY- "-END RSA PRIVATE KEY- "
As far as I know, keytool can only import certificates in PKCS8
format. The "junk" you mentioned may indicate the key is in SSLeay
format. You can use OpenSSL to convert from one format to another.
That said, I'm not aware of _any_ method to import a keypair into a
keystore using keytool; the private key is inaccessible (with respect
to import and export) by design.
You should probably determine whether you actually need the private
key before proceeding. Sounds like you're doing SSL offloading, but
that shouldn't necessarily require using the same keypair on both the
LB and endpoint.
M
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
