RE: [EXT]Re: .intermittent InvocationTargetException errors

[John.E.Gregg]

The NullPointerException is the real problem.


From: Rick Noel 
Sent: Monday, April 15, 2024 10:30 AM
To: Tomcat Users List 
Subject: RE: [EXT]Re: .intermittent InvocationTargetException errors

The complete trace. . . . . . . . . java. lang. reflect. 
InvocationTargetException at java. base/jdk. internal. reflect. 
NativeMethodAccessorImpl. invoke0(Native Method) at java. base/jdk. internal. 
reflect. NativeMethodAccessorImpl. invoke(NativeMethodAccessorImpl. java: 77)


The complete trace.



java.lang.reflect.InvocationTargetException

at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at com.radiovoodoo.display.EvalExpr.evalIdentifier(EvalExpr.java:317)

at com.radiovoodoo.display.EvalExpr.eval(EvalExpr.java:57)

at com.radiovoodoo.display.EvalExpr.eval(EvalExpr.java:30)

at com.radiovoodoo.display.Expr.evaluate(Expr.java:370)

at com.radiovoodoo.display.PropertyTag.evaluate(PropertyTag.java:102)

at com.radiovoodoo.display.PropertyTag.doStartTag(PropertyTag.java:55)

at 
org.apache.jsp.services.system.contesting.contest_005fschedule_005fedit_jsp._jspx_meth_ww_005fproperty_005f5(contest_005fschedule_005fedit_jsp.java:6193)

at 
org.apache.jsp.services.system.contesting.contest_005fschedule_005fedit_jsp._jspService(contest_005fschedule_005fedit_jsp.java:1299)

at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)

at 
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:456)

at 
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:380)

at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:328)

at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)

at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:206)

at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:150)

at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)

at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:175)

at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:150)

at 
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:653)

at 
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:419)

at 
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:340)

at 
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:277)

at 
com.radiovoodoo.action.ServletDispatcher.service(ServletDispatcher.java:177)

at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)

at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:206)

at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:150)

at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)

at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:175)

at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:150)

at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)

at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)

at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)

at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)

at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)

at 
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:673)

at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)

at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)

at 
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)

at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)

at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)

at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1736)

at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorB

RE: Speeding up tomcat

[John.E.Gregg]

Alex


> -Original Message-
> From: Alex Hatcher 
> Sent: Monday, March 25, 2024 12:06 PM
> To: users@tomcat.apache.org
> Subject: Speeding up tomcat
> 
> 
> 
> Java version: 11.0.20
> 
> Tomcat version: 9.0.59
> 
> 
> OS Version: Windows Server 2022 Datacenter Azure Edition
> 
> Azure VM Type and Specs: D8s v3
> 
> 8 CPU 32 GiB Mem
> 
> VM Generation V2
> 
> VM Architecture   x64
> 
> Location   East US Zone 1
> 
> 
> 
> We have a traditional client/server application from a third-party vendor that
> has a couple second delay when accessing certain items (tabs) in their Web UI.
> The delay has been traced down to the webserver, which runs Tomcat.  The
> application and database servers do not appear to have any significant delays.
> 
> 
> 
> When an item is clicked inside the web UI, a call from the web server is made
> to the app and DB server, which come back fairly quickly.  It's at the point
> where data is delivered from the app server to the web server that tomcat on
> the webserver CPU usage spikes.  In reviewing the web server with procmon
> running, we noticed that tomcat is reading a lot of class files during the 
> time
> we are waiting for the task to complete to render the page.   Approximately
> 55,000 (yes 55000) classes read each click.
> 
> 
> 
> The vendor has reviewed this and said it's nothing to worry about, but we
> cannot find any other significant task that tomcat is doing during this wait
> state that a web client experiences.
> 
> 
> 
> We have sql tracing showing microsecond response times.
> 
> 
> 
> Developer console in chrome shows it waiting for 1.9 (Avg) seconds per click.
> 
> 
> 
> We would like to get to the root cause of this slowness, whether it is the
> operating system, Azure VM, webserver or vendor application causing the
> issue.
> 
> 
> 
> 
> 
> 
> Notice: This e-mail message is confidential and is intended only for the use 
> of
> the individual and/or entity identified in the address line of this message. 
> If
> you have received this message in error, or are not the named recipient(s),
> please notify us immediately by telephone (888-479-9111) M_LEGAL_NOTICE

For this level of detail, I would use a profiler like the java flight recorder, 
JProfiler, etc.

Your description of classes being re-read sounds like the code is recreating 
one of the many factory classes that use the java service locator under the 
covers, like DocumentBuilderFactory, etc.  If you can confirm that, I can tell 
you (or the vendor) what to do about it.

Thanks


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: When does Tomcat add and remove threads?

[John.E.Gregg]

All,


> -Original Message-
> From: Christopher Schultz 
> Sent: Tuesday, March 12, 2024 8:31 AM
> To: users@tomcat.apache.org
> Subject: Re: When does Tomcat add and remove threads?
> 
> John,
> 
> On 3/11/24 18:14, john.e.gr...@wellsfargo.com.INVALID wrote:
> > From: Christopher Schultz 
> > Sent: Monday, March 11, 2024 5:09 PM
>  >
> >> On 3/11/24 17:47, john.e.gr...@wellsfargo.com.INVALID wrote:
> >>> I am using Tomcat 9.x.
> >>>
> >>> When does Tomcat add and remove threads from its internal thread
> >>> pool?  I'm talking about the threads with names like
> >>> http-nio-8080-exec-1.  It appears the thread pool is Tomcat's own
> >>> ThreadPoolExecutor but I don't see the exact behavior documented.
> >>> I'm familiar with how java.util.concurrent does it, but it looks
> >>> like Tomcat's version is a little different.
>  >>
> >> Are you looking for a technical explanation with code references, or
> >> a plain-English description of when threads are created and added? >
> > Mostly plain English like the j.u.c. ThreadPoolExecutor Java doc has.
> > What happens when all core threads are in use?  When do tasks go on
> > the queue?  When does core thread + 1 get added?  When do threads get
> > removed?
> Tomcat will create thread pools under two separate circumstances. They are
> related, but behave somewhat differently.
> 
> First, if you declare an  in your server.xml, then a thread pool 
> will be
> created. You can control the number of threads and their retention policy such
> as "keep X spare threads around" and "retire threads after N seconds without
> being used."
> 
> Second, if you declare a  without specifying an "executor", a
> thread pool will be configured for you but you don't really have control over 
> it
> because all those nice configuration options for an  are not 
> available
> on the . If you want to control those settings, use a 
> linked with an . To be clear, if you declare a  without 
> an
> "executor" attribute, your thread pool will be of a fixed size and threads 
> will
> never be released. (I think the thread pool starts small and grows but will
> never shrink.)
> 
> An  is implemented in the StandardThreadExecutor and
> ThreadPoolExecutor classes, which I believe were adaptations of classes from
> java.util.concurrent introduced into Tomcat before java.util.concurrent was
> actually available -- which is why it wasn't used directly in Tomcat. (NB: The
> ThreadPoolExecutor class in Tomcat contains an "@author Doug Lea" tag. The
> Tomcat source is licensed under AL2, the JDK source is licensed under GPL2,
> but the original was released by Doug Lea into the public domain under a CC0
> 1.0 Deed license.)
> 
> The re-sizing occurs in the ThreadPoolExecutor class if you'd like to read 
> it. It is
> not entirely straightforward. You could start by reading the code for the
> runWorker(Worker w) method where, at the end, processWorkerExit is called.
> 
> But since Tomcat's ThreadPoolExecutor is basically Java's ThreadPoolExecutor,
> they work the same.
> 
> -chris
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

I took matters in to my own hands and wrote a test to see what was happening.

Unlike java.util.concurrent.ThreadPoolExecutor, Tomcat's TPE adds threads when 
all core threads are busy.  It only adds tasks to the queue when max threads 
are busy.  I prefer this behavior to the JDK behavior.

Threads are removed when idle for 60 seconds.  This appears to be hard-coded in 
AbstreactEndpoint.createExecutor().

Thanks

Re: When does Tomcat add and remove threads?

[John.E.Gregg]

Chris,


From: Christopher Schultz 
Sent: Monday, March 11, 2024 5:09 PM
To: users@tomcat.apache.org 
Subject: Re: When does Tomcat add and remove threads?

John,

On 3/11/24 17:47, john.e.gr...@wellsfargo.com.INVALID wrote:
> I am using Tomcat 9.x.
>
> When does Tomcat add and remove threads from its internal thread
> pool?  I'm talking about the threads with names like
> http-nio-8080-exec-1.  It appears the thread pool is Tomcat's own
> ThreadPoolExecutor but I don't see the exact behavior documented.
> I'm familiar with how java.util.concurrent does it, but it looks like
> Tomcat's version is a little different.
Are you looking for a technical explanation with code references, or a
plain-English description of when threads are created and added?

-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


Mostly plain English like the j.u.c. ThreadPoolExecutor Java doc has.  What 
happens when all core threads are in use?  When do tasks go on the queue?  When 
does core thread + 1 get added?  When do threads get removed?

Thanks

When does Tomcat add and remove threads?

[John.E.Gregg]

I am using Tomcat 9.x.

When does Tomcat add and remove threads from its internal thread pool?  I'm 
talking about the threads with names like http-nio-8080-exec-1.  It appears the 
thread pool is Tomcat's own ThreadPoolExecutor but I don't see the exact 
behavior documented.  I'm familiar with how java.util.concurrent does it, but 
it looks like Tomcat's version is a little different.

Thanks

RE: Thread Pool Question

[John.E.Gregg]

You have to refer to it in your connector:

https://tomcat.apache.org/tomcat-10.0-doc/config/http.html


> -Original Message-
> From: William Crowell 
> Sent: Tuesday, December 5, 2023 1:39 PM
> To: Tomcat Users List 
> Subject: Re: Thread Pool Question
> 
> I should clarify the ask here...
> 
> I have some long running JDBC queries against Oracle, and I do not want to
> tie up Tomcat's web thread pool with them.  I would only have between 1-10
> threads in this pool.
> 
> Regards,
> 
> William Crowell
> 
> 
> 
> 
> This e-mail may contain information that is privileged or confidential. If you
> are not the intended recipient, please delete the e-mail and any attachments
> and notify us immediately.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: DataSource Connection pool leak

[John.E.Gregg]

Tim,

> -Original Message-
> From: Scott,Tim 
> Sent: Friday, August 25, 2023 3:09 AM
> To: users@tomcat.apache.org
> Subject: DataSource Connection pool leak
> 
> Hi,
> 
> For various diagnostics, I tried Tomcat 9.0.79 recently on a development
> machine. It didn't solve the problem I was experiencing - that was later
> identified as a problem in IntelliJ with remote deployment and is not why I'm
> mailing.
> 
> I tweaked my Tomcat 9.0.79 configuration to start my application manually, as
> IntelliJ refused to deploy remotely.
> 
> This lead to it quickly exhausting its connection pool and then hanging*
> before the application could complete its startup activity.
> * Each request for a connection from the pool timed out. The 
> log
> shows that all 20 connections were allocated.
> 
> I pointed my system back to 9.0.68, and it started up fine.
> 
> I tried all versions I could find back from 9.0.78 through 9.0.71 - and ran 
> into
> the same connection pool problem as 9.0.79 with every version.
> 
> v9.0.70 worked.
> (well, at least didn't exhibit the same problem - the 
> application
> completed its startup activity and was operational).
> 
> I would therefore conclude that something about the way I'm managing my
> database connection pool is preventing 9.0.71 onward from freeing up the
> connections.
> 
> My application uses javax.sql.DataSource with Corretto JDK 11 (11.0.16+8-
> LTS, to be specific).
> 
> It struck me as odd as that means that all releases in the last 8 months have
> this problem for my application. Has anyone else, here, run into a similar
> problem?
> 
> If a pool size of 20 is just too low, I think I'd need to set mine to a few
> hundred to complete the application startup and I'm not willing to try that
> without further insight.
> 
> Thanks,
> Tim
> 
> --
> Tim Scott (he/him/his)
> OCLC * Lead Software Engineer / Technical Product Manager
> 
> 
> cc: IT file

Why does your app need 20 connections just to start up?  That's a bit of a 
rhetorical question, but needing so many connections to start up seems odd to 
me.

Are you using the Tomcat pool or another one?  If it's Tomcat, I think you 
should use some of the abandoned connection settings described here:

https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html

If your app is reserving connections and not releasing them, they will be 
considered abandoned.  With logAbandoned=true, you'll get a stack trace showing 
where the connection was obtained.

Also, you can take thread dumps to see what your threads are doing.  If they 
are actually using the connection, you might see a query in flight.  Then you 
can ask yourself why they're taking so long.

John



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Java Heap Space Error

[John.E.Gregg]

Pratik,

You’re getting the error when executing a DB query.  How many rows does it 
return?  Could it be returning too many rows?

From: pratik.kulka...@shell.com.INVALID 
Sent: Thursday, April 13, 2023 9:55:06 PM
To: users@tomcat.apache.org 
Subject: RE: Java Heap Space Error

Hi All,

Thanks for your quick suggestions.

As Olaf suggested, I tried to set the same values to Xms and Xmx; the 
application immediately crashes after Tomcat restart and I am not able to 
access it. I tried it with different values, but the result is the same. 
Regarding the memory leak, let me check on how to monitor an Oracle Apex 
application running on Tomcat, since this is something I have never done before.

Kevin and John- The heap dump is not generated, but I am looking into 
generating it. I have looked up a tool called visualvm and I am taking it on 
the server. Once I have those, I'll attach them here to see if those can be of 
any help.

Chris - I see. So we already have installed a service and I tried to set the 
environment variable after we got the error. Is there a way for Tomcat to read 
the variables we set after installation? Also, what I meant by "when I access 
the application" is that it worked for a few requests; but as soon as I started 
to launch the pages which have SQL queries being executed to fetch user data, 
the application was crashing with this error. When I set both values for Xms 
and Xmx as Olaf suggested, the application crashed right on Tomcat startup, so 
I couldn't access the URL at all. Also, below is the output of tomcat_stderr,

-
2023-04-14 02:47:53 Apache Commons Daemon procrun stderr initialized.
14-Apr-2023 02:47:55.483 WARNING [main] 
org.apache.tomcat.util.digester.SetPropertiesRule.begin Match 
[Server/Service/Connector] failed to set property [maxSpareThreads] to [75]
14-Apr-2023 02:47:55.563 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server version name:   
Apache Tomcat/9.0.73
14-Apr-2023 02:47:55.563 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server built:  
Feb 27 2023 15:33:40 UTC
14-Apr-2023 02:47:55.563 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server version number: 
9.0.73.0
14-Apr-2023 02:47:55.564 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS Name:   
Windows Server 2016
14-Apr-2023 02:47:55.564 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS Version:
10.0
14-Apr-2023 02:47:55.564 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Architecture:  
amd64
14-Apr-2023 02:47:55.564 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Java Home: 
C:\Program Files\Java\jre1.8.0_361
14-Apr-2023 02:47:55.564 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM Version:   
1.8.0_361-b09
14-Apr-2023 02:47:55.565 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
Oracle Corporation
14-Apr-2023 02:47:55.565 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: 
C:\Program Files\Apache Software Foundation\Tomcat9.0.73
14-Apr-2023 02:47:55.565 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: 
C:\Program Files\Apache Software Foundation\Tomcat9.0.73
14-Apr-2023 02:47:55.566 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Dcatalina.home=C:\Program Files\Apache Software Foundation\Tomcat9.0.73
14-Apr-2023 02:47:55.566 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Dcatalina.base=C:\Program Files\Apache Software Foundation\Tomcat9.0.73
14-Apr-2023 02:47:55.566 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.io.tmpdir=C:\Program Files\Apache Software Foundation\Tomcat9.0.73\temp
14-Apr-2023 02:47:55.567 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
14-Apr-2023 02:47:55.567 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.util.logging.config.file=C:\Program Files\Apache Software 
Foundation\Tomcat9.0.73\conf\logging.properties
14-Apr-2023 02:47:55.567 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
exit
14-Apr-2023 02:47:55.567 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
abort
14-Apr-2023 02:47:55.567 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xms128m
14-Apr-2023 02:47:55.567 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Xmx256m
14-Apr-2023 02:47:55.577 INFO [main] 
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomc

RE: Java Heap Space Error

[John.E.Gregg]

Pratik,


> -Original Message-
> From: pratik.kulka...@shell.com.INVALID
> 
> Sent: Thursday, April 13, 2023 4:36 AM
> To: users@tomcat.apache.org
> Subject: Java Heap Space Error
> 
> Hello,
> 
> This email concerns an error I encountered while using Oracle Apex with
> ORDS 3.0.9 and Tomcat 9. Specifically, I receive a "Java heap space" error
> when accessing the application.
> 
> To troubleshoot the issue, I have tried to increase the maximum heap size of
> the JVM using the -Xmx option in the CATALINA_OPTS environment variable.
> I set this environment variable in the catalina.bat file and restarted the
> Tomcat service, but the error persisted.
> 
> I have also confirmed that the CATALINA_OPTS environment variable is set
> correctly by running echo %CATALINA_OPTS% in the command prompt.
> 
> I should note that I have two versions of Tomcat installed on the system - 8.5
> and 9. Everything works fine on Tomcat 8.5, but the "Java heap space" errors
> are thrown on Tomcat 9.
> 
> I would appreciate any insights or guidance you can provide to help me
> resolve this issue. Below are the details of the software I am using,
> 
> 
>   *   Windows Server 2016 Datacenter
>   *   ORDS 3.0.9
>   *   Tomcat 9.0.73 (which is throwing the error)
>   *   Tomcat 8.5 (which works fine)
>   *   Java 1.8.0_361
> 
> Kind regards,
> Pratik Kulkarni

The most effective thing to do is take heap dumps.  If you enable 
-XX:HeapDumpOnOutOfMemoryError, you'll get one when the app crashes.  If you 
want to be more proactive, you can use jcmd to take them while the app is 
running.

Use something like Eclipse MAT to open the heap dump.  MAT will make some 
educated guesses about what might be the problem.  If that's not enough, look 
at the histogram.  Sometimes the problem will be obvious from a single dump.  
"Oh!  Why do I have 1.21GB of class Foo?"  Sometimes you'll need to take a 
couple of dumps at some interval and compare them.  In that case hopefully 
you'll see a particular class bubbling to the top of the histogram.  Then 
you'll need to figure out why those objects are being kept alive.  Right-click 
the offending class and choose "Merge shortest path to GC roots."  You can also 
choose "List Objects with incoming references."

john

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Need to know about analyzing of thread dump and heap dump

[John.E.Gregg]

> -Original Message-
> From: Suvendu Sekhar Mondal 
> Sent: Tuesday, April 04, 2023 1:54 AM
> To: Tomcat Users List 
> Subject: Re: Need to know about analyzing of thread dump and heap dump
> 
> Hi Koustav,
> 
> On Tue, Apr 4, 2023, 1:28 AM Naha, Koustav 
> wrote:
> 
> > Hi all,
> >
> > Good day.
> >
> > Can someone suggest me some good tools to analyze heap dump and
> thread
> > dumps which we can use in real time production environment.
> > Also, GUI based tools will be a good one to use.
> >
> 
> Pease don't use any online third party tool to analyze heap dumps. Because a
> memory snapshot contains all the details including your secrets :). IMHO, it's
> better to use offline tools like Eclipse MAT, IBM's heap dump analyzer for
> heap dump analysis.
> 
> For thread dumps analysis, in most cases I found that multiple thread dumps
> are required to identify slowness or establish a pattern. I haven't came 
> across
> any thread dumps analyzer which takes multiple thread dumps as input.
> That's why I always go back to IBM thread dumps analyzer.
> 
> Both of these are free to download.
> 
> 
> 
> > Please pour in your 2 cents.
> >
> > Thanks and Regards,
> > Koustav Naha
> >
> >
> > DXC Technology Company -- This message is transmitted to you by or on
> > behalf of DXC Technology Company or one of its affiliates. It is
> > intended exclusively for the addressee. The substance of this message,
> > along with any attachments, may contain proprietary, confidential or
> > privileged information or information that is otherwise legally exempt
> > from disclosure. Any unauthorized review, use, disclosure or
> > distribution is prohibited. If you are not the intended recipient of
> > this message, you are not authorized to read, print, retain, copy or
> > disseminate any part of this message. If you have received this
> > message in error, please destroy and delete all copies and notify the
> > sender by return e-mail. Regardless of content, this e-mail shall not
> > operate to bind DXC Technology Company or any of its affiliates to any
> > order or other contract unless pursuant to explicit written agreement
> > or government initiative expressly permitting the use of e-mail for such
> purpose.
> >


FastThread analyzes more than one dump and compares them.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Database related performance degradation after upgrading from Tomcat 9.0.33 to Tomcat 9.0.69

[John.E.Gregg]

Artur,

> -Original Message-
> From: Artur Tomusiak - Hannon Hill 
> Sent: Tuesday, February 21, 2023 4:31 PM
> To: users@tomcat.apache.org
> Subject: Database related performance degradation after upgrading from
> Tomcat 9.0.33 to Tomcat 9.0.69
> 
> After upgrading from Tomcat 9.0.33 to Tomcat 9.0.69, jobs in our application
> that execute lots of quick database queries end up being visibly slower - 16%
> slower on average in a typical setup where a database is on a local area
> network. Here is additional information we have
> confirmed:
> 
>- This is not specific to operating system (tested on Linux and MacOS)
>- This is not related to database vendors (tested on MySQL and Oracle)
>- This is not related to our software (identical code runs on different
>versions of Tomcat)
>- This is related to database connection latency as opposed to the speed
>of the database or the app - the longer the database latency, the more
>significant the slowdown is. When testing with a local database on the same
>machine, there is no performance hit between the two versions of Tomcat.
>When testing with a database on another network across the Internet (very
>high latency), the job running on Tomcat 9.0.69 is about 50% slower than
>Tomcat 9.0.33.
>- This might be coincidental, but based on the number of queries the job
>executes and the database connection latency, it appears as if each
>database query required 2 additional network trips to the database on
>Tomcat 9.0.69 as compared to Tomcat 9.0.33.
>   - For example, if a job executes about 37,000 queries, and the
>   database connection latency is about 0.15 ms, the job ends up being
> about
>   11 seconds slower on Tomcat 9.0.69 than Tomcat 9.0.33 based on our
> tests.
>   This seems to add up to 2 extra network trips per query because 37,000
>   queries * 0.15 ms/trip * 2 extra trips/query = 11,100 ms = 11.1 s.
>   - Another example is when testing with a db connection over the
>   Internet (25 ms latency) and a job that executes 1,231 queries, that 
> job is
>   more or less 60 seconds slower on Tomcat 9.0.69 than Tomcat
> 9.0.33 based on
>   our tests. Again, if we assumed that there are extra 2 trips to the
>   database per query, this adds up: 1,231 queries * 25 ms/trip * 2 extra
>   trips/query = 61,550 ms = 61.55 s.
> 
> We are suspecting that the slowness comes from around getting a database
> connection from the connection pool, though we spotted an occasional
> slowness around transaction committing as well.
> 
> Here is our database connection configuration in context.xml file:
> 
>name="jdbc/CascadeDS"
> auth="Container"
> type="javax.sql.DataSource"
> username="@{dbusername}"
> password="@{dbpassword}"
> driverClassName="com.mysql.jdbc.Driver"
> 
> url="jdbc:mysql://@{dbhostport}/@{dbname}?useUnicode=true&char
> acterEncoding=UTF-8&useSSL=false"
> maxTotal="250"
> maxIdle="10"
> maxWaitMillis="3000"
> removeAbandonedOnBorrow="true"
> removeAbandonedOnMaintenance="true"
> removeAbandonedTimeout="300"
> logAbandoned="true"
> testOnBorrow="true"
> testOnCreate="true"
>   />
> 
> Is this a known issue? If not, is there any additional information I could
> provide to help troubleshoot or replicate the problem?
> 
> Thank you,
> Artur Tomusiak

I don't know anything about version differences between those versions of 
Tomcat, but I do know a thing or two about JDBC and network round trips.

Assuming the connections are open and in the pool, here are the possible round 
trips that I can think of:

1. Check the health of the connection upon checkout (testOnBorrow=true)
2. Disable autocommit
3. Prepare JDBC statement
4. Execute statement & read initial results
5. Read additional results
6. Commit/roll back transaction
7. Enable auto commit

Statement caching should help with #3.

4 and 5 are a little mushy.  I know with Oracle the first 10 rows by default 
come back in the response to the query.  If there are more results, they will 
require additional round trips unless you've increased the JDBC prefetch size.

7 might surprise you but typically pools store connections with autocommit 
enabled because that is the default state for a connection.  Some pools allow 
you to disable this functionality.  IOW, the autocommit state doesn't change 
unless you explicitly change it.

Also, you need to think about what's going on at the TCP level.  If there are 
enough packets in the response, the DB can't just send them all at once.  It 
might have to pause to wait for the ACKs from the client.  So even if the row 
count is low but the rows are wide, there might be extra pauses in there.  As 
long as the data was the same in your tests, I don't think this should be an 
issue, though.

Hope this helps.

RE: help with high cpu usage

[John.E.Gregg]

Alan,


> -Original Message-
> From: Alan F 
> Sent: Thursday, February 03, 2022 2:51 PM
> To: Tomcat Users List 
> Subject: RE: help with high cpu usage
> 
> My bad here are the correct dumps 10 secs apart
> 
> https://urldefense.com/v3/__https://fastthread.io/my-thread-
> report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0zNzY2ZmZmNy0wZDgyL
> TRhZTItYmE3Mi0zMWQyYTYwN2M1ZjgudHh0&__;!!F9svGWnIaVPGSwU!-
> fDbBdZmwXsUn9QT9ftL8f4PTLoASDZOXCeB4UEo7qKgcIHhANQVT6VMoGvG
> st67ghaaLsg$
> 
> 
> https://urldefense.com/v3/__https://fastthread.io/my-thread-
> report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS00NTE3MWUxNy1jYWRi
> LTRkY2UtODBlNS1lMDk0YTJjNTg1OGEudHh0&__;!!F9svGWnIaVPGSwU!-
> fDbBdZmwXsUn9QT9ftL8f4PTLoASDZOXCeB4UEo7qKgcIHhANQVT6VMoGvG
> st67j-3O5xU$
> 
> 
> 
> 
> -Original Message-
> From: john.e.gr...@wellsfargo.com.INVALID
> 
> Sent: 03 February 2022 19:33
> To: users@tomcat.apache.org
> Subject: RE: help with high cpu usage
> 
> Alan,
> 
> 
> > -Original Message-
> > From: Alan F 
> > Sent: Thursday, February 03, 2022 12:19 PM
> > To: Tomcat Users List 
> > Subject: help with high cpu usage
> >
> > Had some issues today with one prod host. One is fine the other has
> > stuck around 80%Cpu.
> > Ive taken a thread dump from both hosts and would appreciate someone
> > give a once over what it may be before I kill and restart. They are
> > clustered nodes so running identical apps and loadbalanced by a hardware
> balancer.
> >
> > Node1 is ok (relatively!)
> > https://urldefense.com/v3/__https://fastthread.io/my-thread-
> >
> report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS1hODllYzBkZS01OGJjLTQ
> >
> 2ZDQtYWRhNS1kYjkxZjM2NjI1ZTAudHh0&__;!!F9svGWnIaVPGSwU!91fvMc
> > RzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn-mDKbTl_dmUSa0LaAolXlhipl-
> > Fk2pQ$
> >
> >
> > Node 2 still has high CPU usage
> > https://urldefense.com/v3/__https://fastthread.io/my-thread-
> >
> report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0yN2I0YWY4Mi05OWFhL
> >
> TQ3YjYtOGQ2My0wMDMwZjlkNDQzNjMudHh0&__;!!F9svGWnIaVPGSwU!9
> > 1fvMcRzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn-
> > mDKbTl_dmUSa0LaAolXlhipkmLFr3E$
> 
> Taking thread dumps was a good idea but I see very little activity on node 2.
> It looks like two threads are in Hibernate's
> EntityEntryContext.downgradeLocks() method, one is taking the thread
> dump itself, and another is reading a request from a client, I think.  I would
> not expect this to add up to 80% CPU usage unless one of those threads is
> stuck in a loop.  Comparing multiple thread dumps taken 5-10 seconds apart
> would help answer this question.  How much GC is there?  Could these
> Hibernate queries be pulling a huge amount of data from the DB, thus
> causing a lot of GC activity?
> 
> Node 1 looks idle except for the thread taking the thread dump.
> 
> Do you know for sure that it's the Tomcat process that is using the CPU?

Of the 3 dumps from the same node, the first two are 3 hours apart, yet the two 
query threads seem to still be doing the same thing.  (I verified that the 
timestamps in the dumps are different but maybe I made a mistake.)  They are 
both making the Hibernate downgradeLocks call.  This is occurring in the 
context of committing a transaction.  That definitely makes it look like those 
threads are somehow either stuck or are looping.

The 3rd dump has those same two threads actually making queries to Oracle.  So 
within about 10 seconds, we have two threads committing transactions followed 
by making additional queries.

Definitely show these to the developers if you haven't already.

I didn't follow what you said about the GC.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: help with high cpu usage

[John.E.Gregg]

Alan,


> -Original Message-
> From: Alan F 
> Sent: Thursday, February 03, 2022 12:19 PM
> To: Tomcat Users List 
> Subject: help with high cpu usage
> 
> Had some issues today with one prod host. One is fine the other has stuck
> around 80%Cpu.
> Ive taken a thread dump from both hosts and would appreciate someone
> give a once over what it may be before I kill and restart. They are clustered
> nodes so running identical apps and loadbalanced by a hardware balancer.
> 
> Node1 is ok (relatively!)
> https://urldefense.com/v3/__https://fastthread.io/my-thread-
> report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS1hODllYzBkZS01OGJjLTQ
> 2ZDQtYWRhNS1kYjkxZjM2NjI1ZTAudHh0&__;!!F9svGWnIaVPGSwU!91fvMc
> RzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn-mDKbTl_dmUSa0LaAolXlhipl-
> Fk2pQ$
> 
> 
> Node 2 still has high CPU usage
> https://urldefense.com/v3/__https://fastthread.io/my-thread-
> report.jsp?p=c2hhcmVkLzIwMjIvMDIvMy8tLWFwaS0yN2I0YWY4Mi05OWFhL
> TQ3YjYtOGQ2My0wMDMwZjlkNDQzNjMudHh0&__;!!F9svGWnIaVPGSwU!9
> 1fvMcRzcMYr95RrClT-eCrcDNp3fKUDpupDSNtn-
> mDKbTl_dmUSa0LaAolXlhipkmLFr3E$

Taking thread dumps was a good idea but I see very little activity on node 2.  
It looks like two threads are in Hibernate's 
EntityEntryContext.downgradeLocks() method, one is taking the thread dump 
itself, and another is reading a request from a client, I think.  I would not 
expect this to add up to 80% CPU usage unless one of those threads is stuck in 
a loop.  Comparing multiple thread dumps taken 5-10 seconds apart would help 
answer this question.  How much GC is there?  Could these Hibernate queries be 
pulling a huge amount of data from the DB, thus causing a lot of GC activity?

Node 1 looks idle except for the thread taking the thread dump.

Do you know for sure that it's the Tomcat process that is using the CPU?

RE: Tomcat jdbc connections

[John.E.Gregg]

Alan,


> -Original Message-
> From: Alan F 
> Sent: Friday, January 21, 2022 6:53 AM
> To: Tomcat Users List 
> Subject: RE: Tomcat jdbc connections
> 
> Hi Christopher
> 
> Thanks for your time here.
> 
> You mean like, a connection is made, no queries are executed, and then the
> connection is terminated?
> - ANSWER -  DBAs are saying connections last a minute or so and are replaced
> by a new set of connections 1 session each pool.
> 
> 
> Presumably, you have an application running on Tomcat with a JDBC
> connection configured. Are you using Tomcat's built-in pooling, or is your
> application managing its own pooling/connections?
> - ANSWER we are using dbcp2
> 
> 
> Do you have any background tasks (in the JVM) that will run even when
> there is no user activity? Cache-management? Lazy-writes?
> - ANSWER I don't think so.
> 
> 
> Are you using Tomcat's clustering? Are you using a database-backed session
> management or any kind? -
> - ANSWER YES clustered B node down, A node up this Tomcat node is not live
> or receiving any traffic currently in test.
> 
> 
> Are the connections definitely being made from the application itself and/or
> Tomcat? Or do you just see those connections coming from the IP where the
> application is running? - ANSWER im assuming according to server.xml they
> are utlising these declared connections via Tomcat.
> 
> 
> Do you have any background tasks (outside the JVM) that will run even when
> there are no user actions? For example, some monitoring system that pings
> the server to say "can you reach the database?" - ANSWER NO
> 
> 
> O
> 

Could your pool be closing the connections after a set time?  Look at 
timeBetweenEvictionRunsMillis and maxConnLifetimeMillis here:

https://commons.apache.org/proper/commons-dbcp/configuration.html

RE: Max outbound requests setting in Tomcat 9.X

[John.E.Gregg]

Chandra,


> -Original Message-
> From: Gullapalli, Chandra Mouli 
> Sent: Tuesday, September 21, 2021 3:32 PM
> To: users@tomcat.apache.org
> Subject: Max outbound requests setting in Tomcat 9.X
> 
> Hi,
> 
> I know that we can set restrictions on the number of incoming requests to a
> Tomcat server. However if tomcat has to make an outbound http it has only
> max limit of 2 http outbound requests?
> Is there a way to increase the limit of max number of outbound requests
> from a tomcat server to an external url?
> 
> Ubuntu:18.04
> Tomcat: 9.X
> 
> 
> 
> Thank you
> Chandra

Tomcat has nothing to do without outbound connections.

Perhaps you're thinking of Apache HttpClient, which has a default of 2 
connections per route:

https://hc.apache.org/httpcomponents-client-4.5.x/current/tutorial/html/connmgmt.html



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Calculate time to get a connection from JDBC Pool

[John.E.Gregg]

Lasantha,


> -Original Message-
> From: Lasantha Samarakoon 
> Sent: Monday, September 06, 2021 10:22 PM
> To: users@tomcat.apache.org
> Subject: Calculate time to get a connection from JDBC Pool
> 
> Hi all,
> 
> I am working on Tomcat JDBC Pools and I have a requirement that needs to
> calculate the total time it takes to get a connection from the JDBC pool.
> This is to cover the entire connection borrowing process (includes connection
> creation, setting up, validation, etc). The Tomcat version we are using is
> 9.0.34.
> 
> I tried playing around with the interceptors and also walked through the
> respective implementation of Tomcat[1], but couldn't find any extensible
> code segment to catch the before and after points of the get connection
> flow.
> 
> Appreciate your input on any possible solution for this.
> 
> [1]
> https://urldefense.com/v3/__https://github.com/apache/tomcat/tree/9.0.3
> 4/modules/jdbc-pool__;!!F9svGWnIaVPGSwU!6_tpBtEDTx2wg-
> 2SBU2URViWoyhQPdrSNgVO7dErbhcA1-gh-KL_EtXutKh78PSXUt86mkU$
> 
> TIA,
> Lasantha

I don't have an example on hand, but it looks like there is an MBean with the 
property meanBorrowWaitTimeMillis.  Enable JMX like this:

https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html#JMX

RE: More information, Re: Tomcat 8.5.68 failing on takeoff!

[John.E.Gregg]

> -Original Message-
> From: James H. H. Lampert 
> Sent: Monday, August 09, 2021 12:21 PM
> To: users@tomcat.apache.org
> Subject: Re: More information, Re: Tomcat 8.5.68 failing on takeoff!
> 
> On 8/6/21 9:17 AM, Konstantin Kolinko wrote:
> 
> > Try to find what *.jar file in your system contains the above classes.
> >
> > E.g. searching for string "crimson" in *.jar files.
> > That string will be visible in the archive file as it is a name of a 
> > directory.
> 
> I've learned that QShell (a *nix-like shell that was added with Java
> support) on an AS/400 does have both "find" and "grep," so it wasn't quite so
> futile as I thought.
> 
> If I go into QShell, navigate to the JVM home directory, and do a "find . -
> name '*.jar'"
> I get
> 
>   ./jre/lib/ppc64/compressedrefs/jclSC180/vm.jar
>   ./jre/lib/ppc64/default/jclSC180/vm.jar
>   ./jre/lib/ext/db2_classes18.jar
>   ./jre/lib/ext/CmpCrmf.jar
>   ./jre/lib/ext/IBMSecureRandom.jar
>   ./jre/lib/ext/cldrdata.jar
>   ./jre/lib/ext/dnsns.jar
>   ./jre/lib/ext/dtfj.jar
>   ./jre/lib/ext/dtfjview.jar
>   ./jre/lib/ext/gskikm.jar
>   ./jre/lib/ext/healthcenter.jar
>   ./jre/lib/ext/ibmcmsprovider.jar
>   ./jre/lib/ext/ibmjcefips.jar
>   ./jre/lib/ext/ibmjceplus.jar
>   ./jre/lib/ext/ibmjceprovider.jar
>   ./jre/lib/ext/ibmkeycert.jar
>   ./jre/lib/ext/ibmpkcs11impl.jar
>   ./jre/lib/ext/ibmsaslprovider.jar
>   ./jre/lib/ext/ibmxmlcrypto.jar
>   ./jre/lib/ext/ibmxmldsigprovider.jar
>   ./jre/lib/ext/ibmxmlencprovider.jar
>   ./jre/lib/ext/jaccess.jar
>   ./jre/lib/ext/localedata.jar
>   ./jre/lib/ext/nashorn.jar
>   ./jre/lib/ext/traceformat.jar
>   ./jre/lib/ext/xmlencfw.jar
>   ./jre/lib/ext/zipfs.jar
>   ./jre/lib/aggressive.jar
>   ./jre/lib/charsets.jar
>   ./jre/lib/dataaccess.jar
>   ./jre/lib/ddr/j9ddr.jar
>   ./jre/lib/deploy.jar
>   ./jre/lib/ibmcertpathfw.jar
>   ./jre/lib/ibmcertpathprovider.jar
>   ./jre/lib/ibmcfw.jar
>   ./jre/lib/ibmjcefw.jar
>   ./jre/lib/ibmjgssfw.jar
>   ./jre/lib/ibmjgssprovider.jar
>   ./jre/lib/ibmjssefw.jar
>   ./jre/lib/ibmjsseprovider2.jar
>   ./jre/lib/ibmorb.jar
>   ./jre/lib/ibmorbapi.jar
>   ./jre/lib/ibmpkcs.jar
>   ./jre/lib/ibmsaslfw.jar
>   ./jre/lib/javaws.jar
>   ./jre/lib/management-agent.jar
>   ./jre/lib/math.jar
>   ./jre/lib/plugin.jar
>   ./jre/lib/resources.jar
>   ./jre/lib/rt.jar
>   ./jre/lib/se-service.jar
>   ./jre/lib/security/US_export_policy_56bit.jar
>   ./jre/lib/security/local_policy_56bit.jar
>   ./jre/lib/security/policy/limited/US_export_policy.jar
>   ./jre/lib/security/policy/limited/local_policy.jar
>   ./jre/lib/security/policy/unlimited/US_export_policy.jar
>   ./jre/lib/security/policy/unlimited/local_policy.jar
>   ./jre/lib/tools/monitoring-api.jar
>   ./jre/lib/xml.jar
>   ./jre/lib/xmldsigfw.jar
>   ./IBMmisc.jar
>   ./lib/dt.jar
>   ./lib/ibmorbtools.jar
>   ./lib/jconsole.jar
>   ./lib/tools.jar
>   ./lib/IBMi5OSJSSE.jar
> 
> If I do a "find . -name '*crimson*'", I get nothing.
> 
> If I do a "find . -name '*.jar' -exec grep -l 'crimson' {} \;", I also get 
> nothing.
> 
> So unless anybody else has any ideas, I'm once again stuck, at least on this
> angle.
> 
> Mark Thomas wrote:
> > Tomcat 7 doesn't have JASPIC support so you'll never see this issue in
> Tomcat 7.
> 
> . . . to which I replied (seriously, rather than flippantly) "What's a 
> JASPIC?"
> 
> I've finally taken a look at what JASPIC is. Interesting. If it's JASPIC 
> support in
> Tomcat 8 that is throwing exceptions and killing manager under this particular
> Java 8 JVM, is there a way to disable it, at least until the customer has 
> their
> PTFs up to date?
> 
> --
> JHHL
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Check under the Tomcat webapps and lib directories.

RE: More information, Re: Tomcat 8.5.68 failing on takeoff!

[John.E.Gregg]

> -Original Message-
> From: James H. H. Lampert 
> Sent: Friday, August 06, 2021 11:49 AM
> To: Tomcat Users List 
> Subject: Re: More information, Re: Tomcat 8.5.68 failing on takeoff!
> 
> Searching JAR files for "crimson" would likely be an exercise in futility on 
> an
> AS/400.
> 
> --
> JHHL
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

I think it's likely to have "crimson" in the file name.

https://search.maven.org/artifact/crimson/crimson/1.1.3/jar

It's ancient, BTW.

RE: Heap allocations when switching from Tomcat 7 to Tomcat 8

[John.E.Gregg]

James,

> -Original Message-
> From: James H. H. Lampert 
> Sent: Wednesday, June 09, 2021 1:13 PM
> To: Tomcat Users List 
> Subject: Heap allocations when switching from Tomcat 7 to Tomcat 8
> 
> We are beginning to migrate some of our customers from Tomcat 7 to
> Tomcat 8.5.
> 
> Some of them have performance issues even with heap allocations of -
> Xms4096m -Xmx5120m
> 
> Would it be necessary to go even bigger with Tomcat 8.5?
> 
> --
> JHHL
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Somewhere after 7.x there was a change to the Tomcat class loader to not cache 
some resources that previously were cached.  My notes say "classes" are no 
longer cached (but maybe other things loaded via getResourceAsStream() are 
still cached.  I don't remember.)  Ordinarily this would result in less heap 
usage (due to a smaller cache) but...

Did you also change from Java 8 to something else, like 11?  One big difference 
there is that the internal JAXB classes are gone.  If you use JAXB, you need to 
include the corresponding external jars.  With external JAXB jars, Tomcat 9 
generates a lot more garbage than 7.

John

RE: [OT] Request: Encryption requirements for TLS and SSL for Tomcat

[John.E.Gregg]

Emen-Eddine,


> -Original Message-
> From: Christopher Schultz 
> Sent: Wednesday, June 09, 2021 9:08 AM
> To: users@tomcat.apache.org
> Subject: Re: [OT] Request: Encryption requirements for TLS and SSL for
> Tomcat
> 
> Emen-Eddine,
> 
> On 6/8/21 08:10, Emen-Eddine AISSAOUI wrote:
> > Hello,
> >
> > I am contacting you regarding the cipher suite recommandations for TLS
> > and SSL for Tomcat.
> >
> > This is an urgent request for a customer feedback.
> 
> Since this is a customer who is presumably paying YOU for YOUR services, this
> is probably an urgent request for YOU. If your customer(s) want to pay US to
> help them, it may become urgent for US.
> 
> > Could you please tell us which cipher suites are used and necessary
> > and if there is any particular prequesites regarding TLS and SSL
> > encryption for the proper functioning of Tomcat ?
> 
> Tomcat will use a combination of your configuration and system (JVM)
> support to determine which cipher suites will be used. Assuming at least one
> cipher suite is in that set, Tomcat will "work". None are actually necessary.
> 
> -chris
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

If you're looking for actual cipher suite recommendations, I'm not going to 
make any but I will show you some useful resources.

This is a list of the supported Java 11 cipher suites "sorted by order of 
preference."  Hopefully good security is one of their preferences!

https://docs.oracle.com/en/java/javase/11/security/oracle-providers.html#GUID-7093246A-31A3-4304-AC5F-5FB6400405E2

This is another useful site with information on whether a cipher suite is 
recommended or not.

https://ciphersuite.info/cs/

You can cross reference the lists from those two sites.

John


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Need help on ssl handshake logging for audit purpose

[John.E.Gregg]

Raghav,

> -Original Message-
> From: Ragavendhiran Bhiman (rabhiman) 
> Sent: Wednesday, June 09, 2021 6:47 AM
> To: Tomcat Users List 
> Subject: Re: Need help on ssl handshake logging for audit purpose
> 
> Kindly help me on the below.
> 
> Thanks a lot for the help.
> 
> From: Ragavendhiran Bhiman (rabhiman) 
> Date: Tuesday, 8 June 2021 at 7:18 PM
> To: users@tomcat.apache.org 
> Subject: Need help on ssl handshake logging for audit purpose Hi All,
> 
> In our product we are using jdk8 and tomcat apache latest version. I have
> enabled -Djavax.net.debug=ssl:handshake from jdk side. But I could see the
> handshake logging are coming as hex in the Catalina.out log messages. I want
> to know how to print the message in the proper English format. Is any other
> mistake I am doing?
> Kindly help me in this regard.
> 
> Thanks & Regards,
> Raghav

Can you provide an example?  When I use that same debug flag, the only hex I 
see is for binary content, such as the content of a cert.

John

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: [OT] web app big memory usage?

[John.E.Gregg]

Cris,


> -Original Message-
> From: Berneburg, Cris J. - US 
> Sent: Thursday, May 27, 2021 2:24 PM
> To: users@tomcat.apache.org
> Subject: [OT] web app big memory usage?
> 
> Hi Folks  :-)
> 
> One of our web apps is using a "lot" of memory, specifically a big user query.
> We'd like to find out why.
> 
> The Tomcat Web Application Manager Find leaks button said that "No web
> applications appear to have triggered a memory leak on stop, reload or
> undeploy."
> 
> Tomcat Manager Server Status shows that 1.7GB (82%) of G1 Old Gen space
> is being used that has not been recycled yet.
> 
> I grabbed a heap dump and used Eclipse Memory Analyzer, and it shows that
> only 94MB of memory is being used when G1 Old Gen space used 1.8GB.
> MAT seems to be looking only at the active objects, not the discarded ones.
> IOW, we're looking at what the app is doing ATM, not what it already did.
> 
> I want to explore the 1.7GB garbage pile to see what's being thrown away,
> not what things are still being used, to determine wastefulness.
> 
> 1. Is there a way to analyze uncollected garbage?
> 
> 2. Is that a reasonable way to identify potential memory usage problems?
> 
> Some technical specifics:
> * TC 8.5.63
> * Java 1.8.0_291
> * AWS EC2 instance.
> * Windows Server 2016.
> * Instance started as Windows Service.
> * There are other TC instances on the same server.
> * Each TC instance has multiple apps.
> 
> Thanks for reading this far.  :-)
> 
> --
> Cris Berneburg
> CACI Senior Software Engineer
> 
> 
> 
> 
> This electronic message contains information from CACI International Inc or
> subsidiary companies, which may be company sensitive, proprietary,
> privileged or otherwise protected from disclosure. The information is
> intended to be used solely by the recipient(s) named above. If you are not
> an intended recipient, be aware that any review, disclosure, copying,
> distribution or use of this transmission or its contents is prohibited. If you
> have received this transmission in error, please notify the sender
> immediately.
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

MAT has an option  to "Keep unreachable options."  It's under preferences.

It sounds like you don't have an actual leak, just high allocation/GC.  My 
favorite tool for this is to use the Java Flight Recorder and analyze it with 
Java Mission Control.

John


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat SSL stops working after an undetermined amount of time

[John.E.Gregg]

It's "ssl,handshake."


> -Original Message-
> From: Ezsra McDonald 
> Sent: Thursday, May 20, 2021 10:43 AM
> To: Tomcat Users List 
> Subject: Re: Tomcat SSL stops working after an undetermined amount of
> time
> 
> Mark,
> 
> Thanks for your response.
> 
> I did not see anything in the logs. This morning I added '
> -Djava.net.debug=handshake' to my configuration. I did not see any SSL
> debug information in my logs. Perhaps I did this wrong or need to use a
> different argument?
> 
> I expected the debug to be in the access log. Should I be looking elsewhere?
> I also checked other logs that had timestamps for after the instance was
> restarted.
> 
> -- Ez
> 
> On Thu, May 20, 2021 at 3:05 AM Mark Thomas  wrote:
> 
> > On 19/05/2021 20:42, Ezsra McDonald wrote:
> > > Environment:
> > > OS: CentOS 7
> > > Apache: apache-tomcat-8.5.65
> > > Java: jdk1.8.0_281
> > >
> > > Greetings,
> > >
> > > I recently enabled SSL on my Tomcat server HTTP connectors.
> > > Something odd is happening. After some undetermined amount of time
> > > the connector stops responding appropriately to requests. My browser
> > > returns the following
> > > message:
> > >
> > > "An error occurred during a connection to target.host.com:8080. SSL
> > > received a malformed Alert record.
> > >
> > > Error code: SSL_ERROR_RX_MALFORMED_ALERT "
> > > I do not see anything in the logs to clue me in on what is happening.
> > >
> > > I have the following configured for the connector.
> > >  > > port="${http.port}"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > maxThreads="50" enableLookups="false" acceptCount="100"
> > > server="Apache"
> > > SSLEnabled="true" scheme="https" secure="true"
> > > clientAuth="false" sslProtocol="TLSv1.2"
> > > keystoreFile="/opt/tomcat/ssl/tomcat_ssl.jks"
> > > keyAlias="tomcat"
> > > keystorePass="**"
> > > connectionTimeout="2"/>
> > >
> > > When I restart the instance everything works fine for a while.
> > > Later,
> > when
> > > I try to look at the tomcat manager, SSL is no longer functioning
> > properly.
> > >
> > > Any assistance would be appreciated.
> >
> > Anything in the access logs?
> >
> > Enable TLS debug logging in the JVM Tomcat is using. You'll get a lot
> > of data but you'll be able to see exactly what is happening.
> >
> > Mark
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat 9: Application in not starting with TrustStore attributes

[John.E.Gregg]

Manish,

> -Original Message-
> From: Palod, Manish 
> Sent: Tuesday, April 20, 2021 8:37 AM
> To: users@tomcat.apache.org
> Subject: Tomcat 9: Application in not starting with TrustStore attributes
> 
> Hi,
> 
> We are in process of upgrading Tomcat 7 to Tomcat 9 and stuck with Trust
> store settings for Client certificate, following is the connector setting:
> 
>  protocol="org.apache.coyote.http11.Http11NioProtocol"
>maxThreads="150" SSLEnabled="true" scheme="https"
> secure="true" compression="on"
> compressibleMimeType="text/html,text/xml,text/plain,text/javascript,text/
> css,application/x-javascript,application/javascript"
>address="0.0.0.0"
>maxPostSize="10485760"
>URIEncoding="UTF-8" server=" ">
>  truststorePassword="${tomcat.bind.truststorepass}" truststoreType="jks"
> ciphers="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
> certificateVerification="optional" sslProtocol="TLS"
> protocols="TLSv1.2">
>  certificateKeystorePassword ="${tomcat.bind.keystorepass}"
>  type="RSA" />
> 
> 
> 
> Application is working properly when truststoreFile, truststorePassword and
> truststoreType attributes are not defined in SSLHostConfig, when these
> attributes are defined, we are getting following errors at Tomcat start:
> The same configuration parameters are working fine with Tomcat 7. Store
> has 1 valid certificate and rechecked that with keytool with password and
> able to list the certificate.
> Parametrized values are replaced with actual value and that part is working
> fine.
> 
> 
> INFO: Initializing ProtocolHandler ["http-nio-0.0.0.0-80"] Apr 20, 2021 
> 6:59:31
> PM org.apache.coyote.AbstractProtocol init
> INFO: Initializing ProtocolHandler ["https-jsse-nio-0.0.0.0-443"] Apr 20, 2021
> 6:59:31 PM org.apache.catalina.util.LifecycleBase handleSubClassException
> SEVERE: Failed to initialize component [Connector[HTTP/1.1-443]]
> org.apache.catalina.LifecycleException: Protocol handler initialization failed
> at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:1049)
> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
> org.apache.catalina.core.StandardService.initInternal(StandardService.java:5
> 58)
> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
> org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:10
> 45)
> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at org.apache.catalina.startup.Catalina.load(Catalina.java:724)
> at org.apache.catalina.startup.Catalina.load(Catalina.java:746)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472)
> at
> com.intruvert.common.utility.startup.StartupChecks.main(StartupChecks.jav
> a:140)
> Caused by: java.lang.IllegalArgumentException: the trustAnchors parameter
> must be non-empty
> at
> org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(Abstract
> JsseEndpoint.java:99)
> at
> org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEn
> dpoint.java:71)
> at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:246)
> at
> org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndp
> oint.java:1193)
> at
> org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1206
> )
> at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:597)
> at
> org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protoc
> ol.java:80)
> at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:1046)
> ... 14 more
> Caused by: java.security.InvalidAlgorithmParameterException: the
> trustAnchors parameter must be non-empty
> at
> java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200
> )
> at java.security.cert.PKIXParameters.(PKIXParameters.java:157)
> at
> java.security.cert.PKIXBuilderParameters.(PKIXBuilderParameters.java:
> 130)
> at
> org.apache.tomcat.util.net.SSLUtilBase.getParameters(SSLUtilBase.java:501)
> at
> org.apache.tomcat.util.net.SSLUtilBase.getTrustManagers(SSLUtilBase.java:4
> 32)
> at
> org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:2
> 46)
> at
> org.apache.tomcat.util.ne

RE: What exactly is "tenured-SOA"?

[John.E.Gregg]

James,


> -Original Message-
> From: James H. H. Lampert 
> Sent: Thursday, October 22, 2020 2:37 PM
> To: Tomcat Users List 
> Subject: What exactly is "tenured-SOA"?
> 
> In at least two of our Tomcat installations, the Server Status page of Manager
> is showing "tenured-SOA" around 3G, while the other pools are showing
> much lower. What exactly *is* "tenured-SOA," and should this be cause for
> alarm?
> 
> --
> JHHL
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org


The IBM J9 garbage collector splits the old (tenured) generation into soa and 
loa.  That's "small object area" and "large object area."  Small objects are 
moved to soa and large objects to loa.  In my very limited experience with J9, 
most of the action is in soa.  3GB alone doesn't mean much of anything.  It's 
not good or bad.  What really matters is how fast it grows because full GCs are 
more expensive than young GCs.  In general, the heap regions should be sized to 
ensure objects "die young," meaning they should die before they get promoted to 
the old gen.

RE: SSL debug?

[John.E.Gregg]

James,

> -Original Message-
> From: James H. H. Lampert 
> Sent: Tuesday, September 08, 2020 2:13 PM
> To: Tomcat Users List 
> Subject: Re: SSL debug?
> 
> I'm finally back on this problem.
> 
> >> We are once again having SSL difficulties with our webapp connecting
> >> with an outside web service: the java.security override that had
> >> solved the problem in the past (specifically, removing "DESede" from
> >> the "jdk.tls.disabledAlgorithms" in an override file) is now failing
> >> with certain Java 8 JVMs on AS/400s.
> 
> More specifically, the call is to
>  https://maps.googleapis.com/maps/api/geocode
> 
> Last month, I got a suggestion (on the Midrange Java List, but endorsed
> when I asked about it here) of adding "-Djavax.net.debug=ssl:handshake"
> to catalina.sh.
> 
> Today, I finally tried it. It took several tries before I got anything at 
> all, but
> when I did, I got 678k of information. Any suggestions on where my
> proverbial needle would be in that haystack, and what it would look like?
> 
> Also today, I tried the "sledgehammer" approach of editing the JVM's own
> java.security instead of merely overriding it. No difference.
> 
> --
> JHHL
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

I don't remember the precise problem, but verbose SSL will tell you what trust 
store and key store you're using, among other things.  You'll see something 
like this:

keyStore is: /path/to/keystore
...
trustStore is: /path/to/truststore
...

Are those the right files?

After the trustStore path, you should then see a list of all the CAs in the 
trust store.

Ideally, you'd do this on a quiet system.  As you've seen, it produces a ton of 
output and there is no way to distinguish messages from one thread vs another.

RE: Tomcat 8.5.(x > 5) & SSL Connections (sun.security.provider.certpath.SunCertPathBuilderException)

[John.E.Gregg]

> -Original Message-
> From: Mark Thomas 
> Sent: Sunday, August 09, 2020 2:19 PM
> To: Tomcat Users List 
> Subject: Re: Tomcat 8.5.(x > 5) & SSL Connections
> (sun.security.provider.certpath.SunCertPathBuilderException)
> 
> On August 8, 2020 6:59:23 PM UTC, David Filip  wrote:
> >Hello Everyone!
> >
> >I spent a large part of yesterday and this morning trying to debug an
> >SSL problem on Tomcat 8.5.57 to no avail.  I've seen some discussion on
> >either this problem or something related back in 2016, but wanted to
> >confirm what the "correct" solution might be, because I got lost in the
> >threads.
> >
> >I never had this problem with Tomcat 7.0.x, but it started once I
> >upgraded to 8.5.57 (same application code), and it is related to making
> >outgoing SSL connections to web services.  And this is NOT related to a
> >self-signed, but to a commercial (GoDaddy) SSL certificate, albeit on a
> >server that I also run in the cloud.
> >
> >The exception is being thrown when trying to connect to an SSL
> >protected web service is:
> >
> >sun.security.validator.ValidatorException: PKIX path building failed:
> >sun.security.provider.certpath.SunCertPathBuilderException: unable to
> >find valid certification path to requested target
> >
> >although the exact same code worked (and still works on other servers)
> >reliably under Tomcat 7.0.x for several years.
> >
> >Now, here is the weird part: after Google'ing around, I thought the
> >problem might be that Tomcat 8.5.5 and later -- at least this is the
> >gist that I got -- no longer finds the 'default' Java certificate store
> >(cacerts), so I added the following to /bin/catalina.sh (running on a
> >Mac 10.14 / Mojave):
> >
> >export
> >-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.8.0_1
> >21.jdk/Contents/Home/jre/lib/security/cacerts
> >
> >The weird part is that this appeared to fix the problem, so I thought I
> >was done.  Then, I rebooted, and the problem re-appeared!
> >
> >I stopped and started Tomcat, and the problem was resolved again.  I
> >rebooted again, and the problem re-appeared.
> >
> >Previously, when it worked, I refreshed the page several times, and it
> >kept working.  When it doesn't work, if I keep refreshing the page, it
> >continues to throw the exception.
> >
> >Does this mean that some "worker threads" can find the certificate
> >store, and others can't?  Or am I going down the wrong rabbit hole?
> >
> >So, any idea?  The intermittent nature is driving me crazy!
> >
> >And I have can reproduce the problem on two separate servers (both Mac
> >10.14 / Mojave, both Java 1.8.0), one (new server) running 8.5.57 and
> >one (slightly older server) running 8.5.35.  But again, I have several
> >7.0.x instances where I've never seen this problem before.
> >
> >Also, the generic 'SSLPoke' always connects to the service, and it
> >appears that if I run (mostly) the same code from the command line
> >outside of Tomcat (javac / java) it always works.  And if I paste the
> >web service URL into Safari or Chrome, it always works.  And if I use
> >the web service URL with curl (just for good measure), it always works.
> > So it only seems to fall under Tomcat 8.5.x.
> >
> >Thanks in advance for any guidance, as I'm running out of things to
> >Google and try.
> >
> >Regards,
> >
> >Dave.
> 
> Tomcat has zero involvement in outgoing TLS connections.
> 
> If the code works in a standalone Java app, it will work in a Servlet assuming
> the code is thread safe (I don't see why it wouldn't be but worth double
> checking any library being used) and configuration information is accessible.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

I agree with Mark.

This is probably an issue with the inability to find a CA in your trust store 
that matches the cert the server is presenting.  I would guess the different 
versions of Tomcat are looking at different trust stores, which are not 
included with Tomcat.

Probably the best thing to do would be to run with -Djavax.net.debug=ssl.  This 
will tell you what trust store is being used and what's in it.  It says right 
near the beginning of the output "trustStore is: /path/to/trust/store."  It 
will also show you the cert that the server is presenting.  Look below 
"ServerHello."  It produces a ton of output and is almost unusable if more than 
one request is going on at one time however, so it's best if you can run it on 
a "quiet" system.  Output will go to catalina.out.

RE: [OT] Trying to determine the minimum heap required for an operation

[John.E.Gregg]

Chris,
> John,
> 
> On 7/6/20 11:48, john.e.gr...@wellsfargo.com.INVALID wrote:
> > Chris,
> >
> >
> >> -Original Message- From: Christopher Schultz
> >>  Sent: Monday, July 06, 2020 10:21 AM
> >> To: Tomcat Users List  Subject: [OT] Trying
> >> to determine the minimum heap required for an operation
> >>
> > All,
> >
> > Definitely off-topic, but it's the kind of weird thing someone here
> > might have experience with.
> >
> > I have an offline operation I'm considering bringing "inside" my
> > web-based application. My only concern is memory usage: it requires
> > that a bunch of data be loaded from a db into memory and then
> > analyzed. It doesn't take long to execute -- maybe 10 seconds or so,
> > so the memory can be released back to the rest of the application.
> >
> > I've instrumented the command-line process with a thread which runs
> > every .5sec and captures the used-memory, maintaining a high-water
> > mark and reporting it after the whole operation is done. The first
> > time I ran it (with no specific JVM memory-related settings), it
> > reported that the high-water mark was ~450MiB.
> >
> > I figured that was higher than necessary, and probably just
> > represented a lazy GC with loads of memory, so I constrained the
> > process using -Xmx64M. That resulted in a 16MiB high-water mark. I
> > tried again with -Xmx8M and the high-water mark became 5MiB.
> >
> > Is there a particularly good way to force the GC to be as aggressive
> > as possible to see how low I can go, or should I just keep
> > playing-around with the -Xmx setting.
> >
> > Another option is to serialize my in-memory structure to the disk to
> > get a sense of the size in-memory, though it's really not the same --
> > it will at least get me in the right ballpark.
> >
> > Any suggestions?
> >
> > Thanks, -chris
> >>
> >> -
> >>
> >>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
> > I guess I’m that person with the weird experiences.
> >
> > Is memory or CPU in short supply?  If not, I don't think you'll have
> a problem.  This isn't 1997 anymore.  I do think you should run a realistic 
> load
> test, however.
> 
> No specific problem exists, but this is a multi-user web application.
> Usually somewhere around 500 - 1000 users logged-in at once. Session size is
> typically quite low -- only a handful of small objects present with lots of
> sharing of "large" objects and structures. Heap size is set to max 1GiB on 
> each
> server and memory usage shows a beautiful sawtooth pattern hovering
> around ~400MiB for days at a time.
> 
> I will certainly limit the number of these operations that can occur at once,
> and they should be relatively rare. My test example was using a small data
> set, but the size of the data-set varies wildly with the client, so I have to 
> be
> careful for the larger ones.
> 
> Busting the heap isn't something I'd like to have happen.
> 
> > To me the most important GC metric is time spent per
> minute/hour/etc.  The next most important metric is individual pause
> durations.  Through testing you'll see what those numbers are.  I work with
> some large apps that have multi-GB heaps and it's rare to see GC time being
> over 1-2%.  IOW, 600-1200ms per minute.  Often it's a fraction of a percent.
> With those small numbers you're talking about, I don't think you'll have any
> trouble in this area unless the server is very heavily loaded.
> 
> Actually, I'm not super concerned about performance of the GC itself.
> I was just wondering if there was a way to ask the JVM "if you *had* to
> accomplish this task with the smallest possible heap, what would it be?"
> 
> > Be sure to enable verbose GC.  In java 8, it's something like
> > this:
> >
> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps
> - -Xloggc:/path/to/gc.log
> >
> > Run tests with and without the changes.  You can analyze the GC output
> > with tools like GCEasy and GCViewer.
> Sure.
> 
> Again, I'm more concerned with the overhead that will be required for a
> particular operation, so I can predict when running such an operation might
> end up endangering the application server's heap -- and therefore the
> logged-in users.
> 
> Theoretically, if the thread hits a heap-full error, the thread will 
> experience
> an OOME, release it's (temporary) large object tree, and the GC will be able
> to recover, but after an OOME it's never a great plan to trust the JVM for
> very long.
> 
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
> 
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DTR4ACgkQHPApP
> 6U8
> pFiAkQ//cWQ/CL35LJcRIervUhnByPXg/TN1MhfOl66zXx4upJcIpPXgBuIkigbe
> 9d9y/jFnRCyHsFodSEsjtT/q2CxD7k30DIAwRrTaGxzrz60QlD/+t8l3getT9xot
> s0bAxvlpjZTvvhTtpAAv9hkSwJuMxxECksbqmYXaO/rtoBu/N9R8MCjPz4cihTa
> B
> dLZ

RE: [OT] Trying to determine the minimum heap required for an operation

[John.E.Gregg]

Chris,


> -Original Message-
> From: Christopher Schultz 
> Sent: Monday, July 06, 2020 10:21 AM
> To: Tomcat Users List 
> Subject: [OT] Trying to determine the minimum heap required for an
> operation
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> All,
> 
> Definitely off-topic, but it's the kind of weird thing someone here might have
> experience with.
> 
> I have an offline operation I'm considering bringing "inside" my web-based
> application. My only concern is memory usage: it requires that a bunch of
> data be loaded from a db into memory and then analyzed. It doesn't take
> long to execute -- maybe 10 seconds or so, so the memory can be released
> back to the rest of the application.
> 
> I've instrumented the command-line process with a thread which runs every
> .5sec and captures the used-memory, maintaining a high-water mark and
> reporting it after the whole operation is done. The first time I ran it (with 
> no
> specific JVM memory-related settings), it reported that the high-water mark
> was ~450MiB.
> 
> I figured that was higher than necessary, and probably just represented a
> lazy GC with loads of memory, so I constrained the process using -Xmx64M.
> That resulted in a 16MiB high-water mark. I tried again with -Xmx8M and the
> high-water mark became 5MiB.
> 
> Is there a particularly good way to force the GC to be as aggressive as
> possible to see how low I can go, or should I just keep playing-around with
> the -Xmx setting.
> 
> Another option is to serialize my in-memory structure to the disk to get a
> sense of the size in-memory, though it's really not the same -- it will at 
> least
> get me in the right ballpark.
> 
> Any suggestions?
> 
> Thanks,
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
> 
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl8DQT8ACgkQHPApP
> 6U8
> pFhvJA/+N8CfjjWvBwkXSpAW6gbozyqgcxx0zt5z9TEbC4viCZNAQchlh0WE1jx
> F
> dQL2NS6138VNOn45QfVLru7jcVQdk6loRSK4Dxe02neAD6sEwe0v3/zsuu7CDu
> 4x
> Ln3fwohp+5YNxHAUGc4ssGtw8cilShSSLnJCHwG3mxA+grxg6QvVRVqCxV0b+
> sCE
> ocH0MirON5G7b7zETZohtm5lPcghwDy5SBQ4fVo3mLDjUGR8woGr8SL820pQ3
> BuY
> rjGrJ7SHxq+rVnhOrtX6c4YdEebhjR963385kwPf1ND0GoeCp8Yk/LgySxBRPAbh
> 2Kt0UTlbK7wYSDii6kVag1Ayrt5gCyUSrHndvVIl6SdI5gLWfZDbTB3+fvHNg+k5
> x/+Xx/YPvDbXv+b7CtO663uIKV+24iaVq94W+0NVacp3P0YmAmK1CZ9ggs7HQ
> /SC
> uu3R1wRo4yp7eWszhgfpwPHJBvb9Krtfsr8P6rhs5Ry03pzblkmzzTRCvsE85UEZ
> 96RN1OGx2YfPEM4+EN9+rxB1hcElLT8V420MAZd9Jx2n8JmJqdZl6DxJ7vgtvKKj
> 0Y60VaC211M7tzq2zZ5Sh3th3X2tePPoeJQH/vYrreM4snlM8Mt22eQ7jVFri4bY
> F+mu+8DGP3csWmY16nZ0SQ+ZDUS4E9yEplOHq1YKKyHSYGHjn88=
> =u12k
> -END PGP SIGNATURE-
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org


I guess I’m that person with the weird experiences.

Is memory or CPU in short supply?  If not, I don't think you'll have a problem. 
 This isn't 1997 anymore.  I do think you should run a realistic load test, 
however.

To me the most important GC metric is time spent per minute/hour/etc.  The next 
most important metric is individual pause durations.  Through testing you'll 
see what those numbers are.  I work with some large apps that have multi-GB 
heaps and it's rare to see GC time being over 1-2%.  IOW, 600-1200ms per 
minute.  Often it's a fraction of a percent.  With those small numbers you're 
talking about, I don't think you'll have any trouble in this area unless the 
server is very heavily loaded.

Be sure to enable verbose GC.  In java 8, it's something like this:

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps 
-Xloggc:/path/to/gc.log

Run tests with and without the changes.  You can analyze the GC output with 
tools like GCEasy and GCViewer.

John

RE: Having trouble with tomcat 7 installation on RHEL 7.8 power pc

[John.E.Gregg]

Sean,


> -Original Message-
> From: Sean Neeley 
> Sent: Wednesday, July 01, 2020 12:26 PM
> To: Tomcat Users List 
> Subject: Re: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
> 
> John, The top two processes are:
> 
>   PID USER  PR  NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
> 19360 tomcat20   0 3302144  42560  18944 R 99.9  0.5  46:02.37 java
> 19363 tomcat20   0 3302144  42560  18944 R 98.4  0.5  45:48.50 VM Thread
> 
> I tried kill -3 on both of them, plus the java process ID I see from a "ps"
> command, several seconds apart.  The only logs created are these, and they
> are still all empty after the kill -3:
> -rw-r--r--   1 tomcat tomcat0 Jul  1 10:22 catalina.2020-07-01.log
> -rw-r--r--   1 tomcat tomcat0 Jul  1 10:22 host-manager.2020-07-01.log
> -rw-r--r--   1 tomcat tomcat0 Jul  1 10:22 localhost.2020-07-01.log
> -rw-r--r--   1 tomcat tomcat0 Jul  1 10:22 manager.2020-07-01.log
> 
> Any other ideas?
> 
> On Wed, Jul 1, 2020 at 12:09 PM 
> wrote:
> 
> > Sean,
> >
> > > -Original Message-
> > > From: Sean Neeley 
> > > Sent: Wednesday, July 01, 2020 11:15 AM
> > > To: users@tomcat.apache.org
> > > Subject: Having trouble with tomcat 7 installation on RHEL 7.8 power
> > > pc
> > >
> > > I just installed tomcat 7 on a Red Hat Enterprise Linux Server 7.8,
> > power pc
> > > system.  As soon as the service starts, the java process uses 100% cpu.
> > > Logs get created in /var/log/tomcat, but they all have size 0 bytes.
> > > I
> > have not
> > > modified the standard configuration (tomcat.conf, server.xml, etc).
> > > The tomcat packages that are installed are:
> > >
> > > tomcat-7.0.76-12.el7_8.noarch
> > > tomcat-jsp-2.2-api-7.0.76-12.el7_8.noarch
> > > tomcat-el-2.2-api-7.0.76-12.el7_8.noarch
> > > tomcat-lib-7.0.76-12.el7_8.noarch
> > > tomcat-servlet-3.0-api-7.0.76-12.el7_8.noarch
> > >
> > > Are there any tricks I can use to troubleshoot what is going wrong?
> > > Without any logs being created, it seems impossible to determine the
> > > problem.  Thanks for helping.
> > >
> > > Sean
> >
> > Use kill -3 to take several thread dumps about 5-10 seconds apart.
> > They will probably end up in catalina.out.
> >
> > You can also use top -H to see what individual threads are using CPU.
> >
> >

Sorry, I don't know much more.

It's interesting that the CPU is being consumed by "VM Thread."  It doesn't 
look like an ordinary Tomcat or application problem.  When I googled "java vm 
thread," I got some interesting results, such as this one:

https://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/hangloop.html

That also describes the scenario where a thread dump doesn't seem to do 
anything.

RE: Having trouble with tomcat 7 installation on RHEL 7.8 power pc

[John.E.Gregg]

Sean,

> -Original Message-
> From: Sean Neeley 
> Sent: Wednesday, July 01, 2020 11:15 AM
> To: users@tomcat.apache.org
> Subject: Having trouble with tomcat 7 installation on RHEL 7.8 power pc
> 
> I just installed tomcat 7 on a Red Hat Enterprise Linux Server 7.8, power pc
> system.  As soon as the service starts, the java process uses 100% cpu.
> Logs get created in /var/log/tomcat, but they all have size 0 bytes.  I have 
> not
> modified the standard configuration (tomcat.conf, server.xml, etc).  The
> tomcat packages that are installed are:
> 
> tomcat-7.0.76-12.el7_8.noarch
> tomcat-jsp-2.2-api-7.0.76-12.el7_8.noarch
> tomcat-el-2.2-api-7.0.76-12.el7_8.noarch
> tomcat-lib-7.0.76-12.el7_8.noarch
> tomcat-servlet-3.0-api-7.0.76-12.el7_8.noarch
> 
> Are there any tricks I can use to troubleshoot what is going wrong?
> Without any logs being created, it seems impossible to determine the
> problem.  Thanks for helping.
> 
> Sean

Use kill -3 to take several thread dumps about 5-10 seconds apart.  They will 
probably end up in catalina.out.

You can also use top -H to see what individual threads are using CPU.

RE: SSL error [EXTERNAL]

[John.E.Gregg]

Shawn,


-Original Message-
From: Beard, Shawn M.  
Sent: Friday, June 26, 2020 11:57 AM
To: Tomcat Users List 
Subject: RE: SSL error [EXTERNAL]

The code is calling a new webservice. It has godaddy as its ca signer. It was 
getting the error before I added those java options. Those java options were my 
attempt to resolve it. Ive also tried adding the godaddy ca certs to java's 
cacert file without those java options. Same result.



Shawn Beard
Sr. Systems Engineer
BTS
+1-515-564-2528

-Original Message-
From: calder 
Sent: Friday, June 26, 2020 11:45 AM
To: Tomcat Users List 
Subject: Re: SSL error [EXTERNAL]

** CAUTION: External message


In Fri, Jun 26, 2020, 10:37 Beard, Shawn M. 
wrote:

> We are running tomcat-7.0.52(old I know) and java 1.7.0_80.
>

yea, BOTH are very old.

When the app makes calls to an external webservice. It keeps throwing this
> error:
>
> javax.net.ssl.SSLException : javax.net.ssl.SSLException:
> java.lang.RuntimeException: Unexpected error:
> java.security.InvalidAlgorithmParameterException: the trustAnchors 
> parameter must be non-empty
>
[1]

> I have this in the java options and have confirmed the proper CA certs 
> for this webservice is in the truststore. Any ideas?
>
-Djavax.net.ssl.trustStore=/path/to/truststore/tomcatTrustStore.jks
> -Djavax.net.ssl.trustStorePassword=
> -Djavax.net.ssl.trustStoreType=jks
>

Did this runtime EVER work?

If yes, "what" changed?



[1]
https://urldefense.com/v3/__https://stackoverflow.com/questions/6784463/error-trustanchors-parameter-must-be-non-empty__;!!Li8W9_Um1Taa!uk48yx6ZQNHjmcqPmjBlJDFCcCWu6HMZu3OI_Yau1oJ4CBGoaFzI0pfKTaIrqOGk$
CONFIDENTIALITY NOTICE: This e-mail and the transmitted documents contain 
private, privileged and confidential information belonging to the sender. The 
information therein is solely for the use of the addressee. If your receipt of 
this transmission has occurred as the result of an error, please immediately 
notify us so we can arrange for the return of the documents. In such 
circumstances, you are advised that you may not disclose, copy, distribute or 
take any other action in reliance on the information transmitted.
B CB  [  
X  ܚX KK[XZ[
 \ \  ][  X  ܚX P X ]
 \X K ܙ B  ܈Y][ۘ[  [X[  K[XZ[
 \ \  Z[ X ]
 \X K ܙ B 

That error message comes from PKIXParameters.setTrustAnchors().  I was able to 
reproduce the problem with an empty trust store.  I also tried a trust store 
with the wrong certs but got a different error.

With -Djavax.net.debug=ssl, you should see output like this:

trustStore is: /path/to/trust.jks
trustStore type is: jks
trustStore provider is: 
the last modified time is: Fri Jun 26 13:27:52 CDT 2020
Reload the trust store
Reload trust certs
Reloaded 1 trust certs
adding as trusted cert:

Followed by a list of certs found in the store.

Is that what's happening in your case?

John

RE: Issue found during migration of Tomcat version 6.0.35 to 8.5.5

[John.E.Gregg]

Sorry for the top post.  I'm having email formatting problems.

Lavitesh,

15-Jun-2020 07:34:45.122 SEVERE [http-nio-7080-exec-10] 
org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for 
servlet [MainControllerServlet] in context with path [/porequest] threw 
exception [Servlet execution threw an exception] with root cause
javax.xml.stream.FactoryConfigurationError: Provider 
com.ctc.wstx.stax.WstxInputFactory not found
at javax.xml.stream.XMLInputFactory.newInstance(Unknown Source)

This is saying it can't find WstxInputFactory, which appears to be the Woodstox 
implementation of the XMLInputFactory abstract class.

If you want to continue using Woodstox, you need to add it to your classpath 
again.

If you don't want to use it, there are a few things to check to find out why 
it's being used.  Look for:

-  System property 
javax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory

-  $JAVA_HOME/lib/stax.properties

-  $JAVA_HOME/lib/jaxp.properties

-  META-INF/services/javax.xml.stream.XMLInputFactory

That last one is actually a file of the same name as the class.  There is no 
traditional file extension.  This file might also be in a 3rd-party lib, too.  
The file contains the name of the implementation to use.

One of those things is probably telling the finder to use that Woodstox class.

If you don't want to use Woodstox anymore, you have to remove or change one of 
those sources.  Removing it just means you'll get the default.

John



From: Lavitesh Verma 
Sent: Monday, June 15, 2020 8:49 AM
To: Tomcat Users List 
Subject: RE: Issue found during migration of Tomcat version 6.0.35 to 8.5.5


Hi,



PFA the complete Stack Trace for the Issue.

Thanks & Regards
Lavitesh Verma
Software Engineering Associate
Amdocs Global SmartOps
[download12]+91.9810157771
OOO - 06/16 - 06/19
[cid:image001.jpg@01D2912B.17505E90]




-Original Message-

From: Jason Wee mailto:peich...@gmail.com>>

Sent: Monday, June 15, 2020 6:49 PM

To: Tomcat Users List mailto:users@tomcat.apache.org>>

Subject: Re: Issue found during migration of Tomcat version 6.0.35 to 8.5.5



guess looks like jar not found, full stacktrace would be helpful.



On Mon, Jun 15, 2020 at 8:17 PM Lavitesh Verma 
mailto:lavitesh.ve...@amdocs.com>>

wrote:



> Hi Team,

>

>

>

> Below are the details of the system and tomcat version

>

> Old tomcat version: *Apache Tomcat/6.0.35*

>

> New tomcat version: *Apache Tomcat/8.5.5*

>

> Operating System: *SunOS *

>

> OS Version: *5.10*

>

> Architecture: *sparcv9*

>

> JVM Version: *1.8.0_101-b13*

>

> Vendor: *Oracle Corporation*

>

>

>

> We are trying to migrate Apache Tomcat version 6.0.35 to 8.5.5.

>

>

>

> We found the issue javax.xml.stream.FactoryConfigurationError:

> Provider com.ctc.wstx.stax.WstxInputFactory not found in localhost logs.

>

>

>

>

>

> Could you please assist on how we could resolve the issue.

>

>

>

> Thanks & Regards

>

> *Lavitesh Verma*

>

> Software Engineering Associate

>

> Amdocs Global SmartOps

>

> [image: download12]+91.9810157771

>

> *OOO - 06/16 - 06/19*

>

> [image: cid:image001.jpg@01D2912B.17505E90]

>

>

>

> *This email and the information contained herein is proprietary and

> confidential and subject to the Amdocs Email Terms of Service, which

> you may review at*

> *https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww

> .amdocs.com%2Fabout%2Femail-terms-of-service*&data=02%7C01%7CLavit

> esh.Verma%40amdocs.com%7Cf5593255a1244392bb8a08d8113145f6%7Cc8eca3ca12

> 7646d59d9da0f2a028920f%7C0%7C0%7C637278250614631760&sdata=%2B6rV%2

> Bbnayil4ZJr7yAuCsl2YyE86CZV19JWANZPz%2BIo%3D&reserved=0

>  .amdocs.com%2Fabout%2Femail-terms-of-service&data=02%7C01%7CLavite

> sh.Verma%40amdocs.com%7Cf5593255a1244392bb8a08d8113145f6%7Cc8eca3ca127

> 646d59d9da0f2a028920f%7C0%7C0%7C637278250614631760&sdata=Wtzp%2BGE

> %2BjEvBwZwExHs1jlNLsN5wxGZvYhHK7SAwfxM%3D&reserved=0>

>
This email and the information contained herein is proprietary and confidential 
and subject to the Amdocs Email Terms of Service, which you may review at 
https://www.amdocs.com/about/email-terms-of-service

RE: Does Tomcat/Java get around the problem of 64K maximum client source ports?

[John.E.Gregg]

Chris, André, Eric, etc,

A few random thoughts.

First, this is really a MySQL driver thing, not a Tomcat thing.  If you want to 
know why the driver does what it does, look at the driver source.  There is 
actually a Socket constructor that allows you to specify the local port.  I 
don't know why you would do that, but it's there.

Second, it does appear to be possible for there to be multiple connections 
using the same source port as long as they go to different destinations.  Just 
read through the various links that people have provided.  Honestly it blows my 
mind that that's even possible, but that just shows how little I know about 
networking.  How does the client OS route return packets correctly if they have 
the same source port?

John

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: JVM job for Tomcat taking lots and lots of CPU

[John.E.Gregg]

James,



> -Original Message-
> From: James H. H. Lampert 
> Sent: Wednesday, February 12, 2020 12:47 PM
> To: Tomcat Users List ; Java 400 List  l...@lists.midrange.com>
> Subject: Re: JVM job for Tomcat taking lots and lots of CPU
> 
> I've got some more detailed GC stats. The full report runs 600 pages for just
> the last 300 GC cycles, so I've just included three cycles worth of
> data:
> 
> > GC Cycle Number  :   1801
> > Basic GC Cycle Information:
> >   Current GC cycle time  . . . . . . . :399
> >   GC reason  . . . . . . . . . . . . . :   Allocation Failure
> >   GC area  . . . . . . . . . . . . . . :   None specified
> >   GC compaction reason . . . . . . . . :   None specified
> >   Number of internal cycles  . . . . . :  1
> >   Time spent in excessive GC time  . . :  0
> >   Number of objects moved  . . . . . . :  0
> >   Amount of space consumed by moved
> > objects  . . . . . . . . . . . . . :  0
> >   Number of classes unloaded . . . . . :  0
> > GC Time Information:
> >   Cycle start time . . . . . . . . . . :02/12/20 11:58:08.104
> >   Cycle end time . . . . . . . . . . . :02/12/20 11:58:08.503
> >   Mark start time  . . . . . . . . . . : 0
> >   Mark end time  . . . . . . . . . . . : 0
> >   Sweep start time . . . . . . . . . . : 0
> >   Sweep end time . . . . . . . . . . . : 0
> >   Compact start time . . . . . . . . . : 0
> >   Compact end time . . . . . . . . . . : 0
> > Nursery Area Information:
> >   Free space at start  . . . . . . . . :  0
> >   Allocated space at start . . . . . . :1179648
> >   Total size at start  . . . . . . . . :1179648
> >   Free space at end  . . . . . . . . . :1140219
> >   Allocated space at end . . . . . . . :  39429
> >   Total size at end  . . . . . . . . . :1179648
> > Tenured Area:
> >   Free space at start  . . . . . . . . :1119550
> >   Allocated space at start . . . . . . :1717250
> >   Total size at start  . . . . . . . . :2836800
> >   Free space at end  . . . . . . . . . :1087674
> >   Allocated space at end . . . . . . . :1749126
> >   Total size at end  . . . . . . . . . :2836800
> > Large Object Tenured Area:
> >   Free space at start  . . . . . . . . :  0
> >   Allocated space at start . . . . . . :  0
> >   Total size at start  . . . . . . . . :  0
> >   Free space at end  . . . . . . . . . :  0
> >   Allocated space at end . . . . . . . :  0
> >   Total size at end  . . . . . . . . . :  0
> > Small Object Tenured Area:
> >   Free space at start  . . . . . . . . :1119550
> >   Allocated space at start . . . . . . :1717250
> >   Total size at start  . . . . . . . . :2836800
> >   Free space at end  . . . . . . . . . :1087674
> >   Allocated space at end . . . . . . . :1749126
> >   Total size at end  . . . . . . . . . :2836800
> > Weak Object References:
> >   Number at start  . . . . . . . . . . :  0
> >   Number at end  . . . . . . . . . . . :  0
> >   Number cleared . . . . . . . . . . . :  0
> > Finalizer Object References:
> >   Number at start  . . . . . . . . . . :  0
> >   Number at end  . . . . . . . . . . . :763
> >   Number cleared . . . . . . . . . . . : 4294966533
> > Soft Object References:
> >   Number at start  . . . . . . . . . . :  0
> >   Number at end  . . . . . . . . . . . :  0
> >   Number cleared . . . . . . . . . . . :  0
> > Phantom Object References:
> >   Number at start  . . . . . . . . . . :  0
> >   Number at end  . . . . . . . . . . . :  0
> >   Number cleared . . . . . . . . . . . :  0
> >
> >
> > GC Cycle Number  :   1802
> > Basic GC Cycle Information:
> >   Current GC cycle time  . . . . . . . :181
> >   GC reason  . . . . . . . . . . . . . :   Allocation Failure
> >   GC area  . . . . . . . . . . . . . . :   None specified
> >   GC compaction reason . . . . . . . . :   None specified
> >   Number of internal cycles  . . . . . :  1
> >   Time spent in excessive GC time  . . :  0
> >   Number of objects moved  . . . . . . :  0
> >   Amount of space consumed by moved
> > objects  . . . . . . . . . . . . . :  0
> >   Number of classes unloaded . . . . . :  0
> > GC Time Information:
> >   Cycle start time . . . . . . . . . . :02/12/20 11:58:30.544
> >   Cycle end time . . . . . . . . . . . :02/12/20 11:58:30.726
> >   Mark start time  . . . . . . . . 

RE: JVM job for Tomcat taking lots and lots of CPU

[John.E.Gregg]

James,

> -Original Message-
> From: James H. H. Lampert 
> Sent: Tuesday, February 11, 2020 6:41 PM
> To: Tomcat Users List 
> Subject: JVM job for Tomcat taking lots and lots of CPU
> 
> Ladies and Gentlemen:
> 
> We have a customer installation in which the JVM job for our Tomcat server
> is frequently using massive amounts of CPU.
> 
> It's Tomcat 7.0.67, running on an AS/400, in a 64-bit Java 7 JVM, with -
> Xms3096m and -Xmx5120m JVM arguments.
> 
> GC information on the JVM job shows:
> > Garbage collected heap:
> >   Initial heap size  . . . . . . . . . :  3096.000M
> >   Maximum heap size  . . . . . . . . . :  5120.000M
> >   Current heap size  . . . . . . . . . :  4458.562M
> >   Heap in use  . . . . . . . . . . . . :  1907.673M
> > Other memory:
> >   Internal (break) memory size . . . . :   504.982M
> >   JIT memory size  . . . . . . . . . . :74.000M
> >   Shared classes memory size . . . . . : 0.000M
> > General GC information:
> >   Current GC cycle . . . . . . . . . . :   2184
> >   GC policy type . . . . . . . . . . . : GENCON
> >   Current GC cycle time  . . . . . . . :552
> >   Accumulated GC time  . . . . . . . . :5108241
> 
> It seems to be doing a lot of garbage-collecting.
> 
> Would switching to Java 8 help? Would switching to 7.0.93 help?
> 
> --
> James H. H. Lampert

I haven't worked with java on the AS400.

You said high CPU is the symptom.  Frequent GC is certainly one possible cause, 
but there are others.  If possible, take several thread dumps at short 
intervals (5-10 seconds) and review them to see what they have in common.  If 
you're not a java developer, ask one for help.  On Linux, it's possible to 
correlate the thread ids with process ids obtained from top to see how much CPU 
each one is using/has used.

I can't tell whether GC might be the culprit from the data you provided.  The 
first thing I always look at is the throughput, which is 1 - (gc time/total 
time).  You want that to be as close to 100% as possible.  Take that 
accumulated GC time on the last line and divide by the time the app has been 
running and subtract from 1.  Hopefully that number is up around .98 or .99.  
You also have to keep the time range in mind.  If the app sits idle at night, 
then the total throughput will look good because there were so few GCs during 
that idle time.  However if there is a surge in activity over the last few 
minutes, that number could be very different over that short time range.

I assume "current GC cycles" means that this is the 2184th GC since the JVM 
started.  Ok, but how frequent are they?  I've been busy apps do 5-20 minor GCs 
per minute, so 2000 total isn't a scary number to me.

Does "gencon" mean it's collecting the old generation at that moment?  If there 
are really 2000 of those, I would be mildly concerned.  The generations should 
be sized so that the old generation grows slowly.  Some of the apps I work with 
only do 1 or 2 of those per day.  This doesn't necessarily indicate a bug, 
however.  It might just mean that the young generation needs to be increased.

The fact that your used heap is so much lower than your total heap is a good 
sign.

Is JIT memory size how much space is allocated for compiled code?  How much is 
actually used?  By default, HotSpot allocates 240MB and I regularly see apps 
that use more than 74MB.  I don't know what your JVM does when that fills up.  
HotSpot used to puke but is better behaved now.  If you've filled up the space 
allocated for compiled code, I could definitely see this being a contributor to 
high CPU because of a) code running in interpreted mode and b) the JIT compiler 
having to run.

John

RE: Dates on Linux vs. Windows

[John.E.Gregg]

> -Original Message-
> From: Jerry Malcolm 
> Sent: Tuesday, January 07, 2020 3:14 PM
> To: users@tomcat.apache.org
> Subject: Re: Dates on Linux vs. Windows
> 
> On 1/7/2020 3:09 PM, Michael Osipov wrote:
> > Am 2020-01-07 um 21:58 schrieb Jerry Malcolm:
> >> This may be more of a Java question than Tomcat.  But I'm not sure. I
> >> have the same code, talking to the same MySql Linux (AWS) database.
> >> I read a date column value in a Tomcat app.  After calling
> >> resultSet.getDate(...) I printed the date instance and the getTime()
> >> value:
> >>
> >> On windows: 2019-02-01 154900080
> >>
> >> On linux:   2019-01-31 154897920
> >>
> >> Again this is the SAME line of code in java reading the SAME field in
> >> the SAME database.  Only thing different is Linux/Windows OS.  The
> >> date is supposed to be 2/1/2019 and shows that in phpMyAdmin.
> >>
> >> I've been running on Linux for a few months.  But I don't have an
> >> extensive background in the specifics of Linux.  I'm sure there must
> >> be something that is configured differently.  I'm at a loss. But this
> >> is not a trivial problem.  I do monthly billing. My dates need to be
> >> accurate.
> >
> > Have you verified that you aren't tricked by any timezone issues?
> Probably so.  But how would I know?  I was under the impression that
> java.sql.Date was timezone independent.  Shouldn't it simply convert a
> month/day/year value to the number of milliseconds since the epoch?  How
> would timezone issues affect that?  And if I am 'tricked' how do I
> 'untrick'.  What do I set/change?
> >
> >

Those millisecond values are 6 hours apart, which looks like a timezone issue.  
I happen to be in US Central time, which is 6 hours earlier than UTC in winter.

You're right that System.currentTimeMillis() itself is independent of timezone 
but Date is not.

RE: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib

[John.E.Gregg]

Tony,


> -Original Message-
> From: Rhuberg,Anthony 
> Sent: Thursday, October 10, 2019 5:22 PM
> To: Tomcat Users List 
> Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk usage
> because of repeated opening/closing of jars in WEB-INF/lib
> 
> We are still investigating what specific classloader reads that would trigger
> the repeated reload of the web-inf/lib/*.jars.
> 
> One example we found is the use of
> javax.xml.transform.TransformerFactory.newInstance(). One of its features
> is to determine the implementation by searching for the configuration
> specified by META-INF/services/javax.xml.transform.TransformerFactory in
> jars available to the runtime.
> 

Yep.  I was wondering about that.  I have no idea why this behavior would be 
different in Tomcat 9, though.  Did you change Java versions also?  Is the 
classpath longer?

I also have no idea about the jar modification date changes that you're seeing.

I do have first-hand experience with the performance of 
TransformerFactory.newInstance().  In general, anything that uses the service 
locator under the covers, such as TransformerFactory, DocumentBuilderFactory, 
etc, should be reused if possible.  Those XML factories are thread safe, though 
the things created by the factories are typically not thread safe.  You should 
create it once, configure it once, and use it as many times as necessary.  For 
example, you would create one TransformerFactory, then call newTransformer() as 
many times as necessary.

John

RE: Profiler for Tomcat

[John.E.Gregg]

Michael,


> -Original Message-
> From: Michael Duffy 
> Sent: Tuesday, August 27, 2019 5:47 PM
> To: users@tomcat.apache.org
> Subject: Profiler for Tomcat
> 
> I have searched for a good profiler for Tomcat with little success.
> 
> I am looking for an application that will profile internal memory and
> bandwidth utilized (data transfer rates from Tomcat).
> 
> Any help would be greatly appreciated.
> 
> Thx!

Flight recordings made with Mission Control have pretty useful memory 
allocation data.  I've definitely found some low-hanging fruit with it.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat Server Using 100% CPU

[John.E.Gregg]

Eric,

> -Original Message-
> From: Eric Robinson 
> Sent: Thursday, August 08, 2019 11:53 AM
> To: users@tomcat.apache.org
> Subject: Tomcat Server Using 100% CPU
> 
> We have a farm of VMs, each running multiple instances of tomcat (up to 80
> instances per server). Everything has been running fine for years, but
> recently one server has started nailing the CPU to 100% utilization.
> 
> We have tried:
> 
> 
>   *   Different versions of tomcat and JDK
>   *   Doubling the resources to 16 cores and 56 GB RAM
>   *   Moving the VM to different physical server
>   *   Rebuilding the tomcat instances on a brand new VM using Windows
> Server 2019
>   *   Rebuilding the tomcat instances on a brand new VM using Red Hat
> Enterprise Linux 7.5
> 
> Nothing has worked. No matter where we run the tomcats, they drive CPU
> up to 100%. Meanwhile the other six servers are still running fine. They all
> run the same canned tomcat applications.
> 
> We would appreciate some guidance on getting to the bottom of this
> problem.
> 
> --Eric
> 
> 

Are you sure that it's the Tomcat process that is using the CPU?  If so, take 
several thread dumps about 5 seconds apart.  On Linux, use kill -3 
tomcatpidhere.  They will go to catalina.out.  You'll probably need a developer 
to interpret them.



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

failing fast when the server is overloaded

[John.E.Gregg]

Folks,

I've been using Tomcat for a long time but am new-ish to NIO (Tomcat 8.5.)  It 
seems that one of the big benefits of NIO is decoupling the worker threads from 
the client connections.  I can now have a large number of open connections 
without a corresponding large number of threads.

I know the acceptCount parameter will stop incoming connection requests if the 
server is overloaded.  However, suppose I have 1000 open connections and 100 of 
those connections have a request in flight, using all 100 of the threads I've 
allocated.  Now what happens if the other 900 connections suddenly send me 
requests?  Does acceptCount play any role?  Is there some other mechanism that 
I can use to fail fast (by sending back a 503, for example?)

Thanks

John

RE: Does Tomcat count incoming connections?

[John.E.Gregg]

Chris, 


> -Original Message-
> From: Christopher Schultz 
> Sent: Friday, February 08, 2019 11:48 AM
> To: users@tomcat.apache.org
> Subject: Re: Does Tomcat count incoming connections?
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> John,
> 
> 
> You can get this count from the Connector via JMX.
> 
> Look at the tree under /Catalina/GlobalRequestProcessor/[connector]
> and there is a "requestCount" attribute that can be queried.
> 
> Looks like it's an int value, so it can overflow.
> 
> - -chris

Thanks but the requestCount really does appear to be requests and not 
connections.  I have keepAliveTimeout and maxKeepAliveRequests at their 
defaults (unlimited timeout and 100 requests) and I see the count go up with 
each request, even though my client reports that the connection is getting 
reused.

I'm asking because I need some hard data on whether clients are reusing their 
connections for performance reasons.

John

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Does Tomcat count incoming connections?

[John.E.Gregg]

Hi all,

I'm interested in knowing how many new incoming connections are being created, 
possibly per connector.  Does Tomcat have such a thing?  I assume it would 
involve counting calls to ServerSocketChannel.accept() or something like that.  
I see the GlobalRequestProcessor MBean has a request count, but I would also 
like to know connection count.

Thank you

John

RE: Tomcat memory growth while using TLS

[John.E.Gregg]

Mason, Mark, etc...

> -Original Message-
> From: Mark Thomas 
> Sent: Thursday, January 10, 2019 10:45 AM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat memory growth while using TLS
> 
> On 09/01/2019 14:24, Mark Thomas wrote:
> > On 09/01/2019 10:14, Mark Thomas wrote:
> 
> 
> 
> >> It may be you are seeing the same thing. Or you may have found a
> >> memory leak. The next step would be to use a profiler to see where
> >> the memory is being used.
> >
> > Using NIO + OpenSSL with the settings from the commented out
> > APR/native TLS connector in server.xml, the heap and non-heap memory
> > reported by the profiler reach steady state at fairly low values.
> > However, process memory continues to steadily increase. Those
> > observations are consistent with a memory leak in native code (but they
> are not proof).
> >
> > Next steps are looking at additional logging to see where the memory
> > allocations occur.
> 
> I didn't really get anywhere with that. There was a lot of noise from the JVM
> and the performance hit was several orders of magnitude so there was no
> obvious built up of leaked memory.
> 
> However, there has been some progress.
> 
> I've tested trunk (9.0.x) with:
> - Oracle JDK 1.8.0_192
> - Tomcat Native 1.2.19
> - OpenSSL 1.1.2-dev
> - JMeter (20 threads, /index.jsp, no keep-alive)
> 
> and the commented out APT/native TLS connector in the default server.xml.
> 
> APR/native connector - no leak
> NIO+JSSE - no leak
> NIO+OpenSSL - leak
> 
> Interestingly, the performance of the above was roughly:
> NIO+OpenSSL - 5000 req/s
> NIO+JSSE- 4000 req/s
> APR/native  -  400 req/s
> 
> That difference is so large I wonder if I was doing something wrong.
> Something to come back to later.
> 
> Given than only NIO+OpenSSL is affected (or at least it is significantly more
> affected than the other combinations) I'm going to take another look at the
> code focussing on the parts that are specific to that configuration.
> 
> Mark
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

The first troubleshooting step should be to take a heap dump and analyze it 
with a tool like Eclipse MAT.

I don't know anything about the OpenSSL implementation, but it probably has a 
session cache.  Is there an upper limit on the cache size?  The JSSE cache size 
is unlimited by default, I think, and objects only expire from the cache after 
1 day.

When a client connects, if it already has a session id from a previous 
handshake, it will ask the server to resume the session.  If it happens to 
connect to the same server, it will probably get to reuse the existing (cached) 
session.  If not, then a new session is created and cached.  If you have a lot 
of servers behind a load balancer with no stickiness, then the session reuse 
will not be high.  For example, if you have 2 servers, you're only going to get 
50% reuse.  If you have 100 servers, only 1% reuse!  )-:  The lower the session 
reuse rate, the more orphaned sessions there are sitting on the server.

Mark, in your test if you were connecting from the same client to the same 
server over and over, you probably used the same session over and over, so you 
wouldn't see much session cache growth, if any.  I'm not sure of a way to tell 
the client or server to not reuse sessions for testing purposes.  Even 
round-robining across only 2 servers would be more realistic because the first 
handshake will result in session 123, then the second (to the other server) 
would result in session 456 (because the second server doesn't know anything 
about session 123,) then the third might be to the same server as the first, 
but by then the client has forgotten about session 123 and tries to resume 456, 
which doesn't exist on the first server, so you now get session 789 created, 
and so forth.

John

RE: 2 Factor Authentication Tomcat 7

[John.E.Gregg]

Will,


> -Original Message-
> From: Will Nordmeyer 
> Sent: Tuesday, October 23, 2018 9:45 AM
> To: Tomcat Users List 
> Subject: 2 Factor Authentication Tomcat 7
> 
> I'm currently running Tomcat 7 (will likely migrate to 8 or 9 in the next 
> year).  I
> tried working with Oracle on this with no success.
> 
> We have an Oracle Database connection defined within our web.xml (see
> below).  We need to convert to using 2 Factor (certificate?) based
> Authentication.
> 
> How do we convert from our embedded username password to 2FA
> 
> 
> type
> SIMPLE
> 
> 
> 
> datasource
>  
> 
> 
> 
> driver
> oracle.jdbc.OracleDriver
> 
> 
> 
> url
> jdbc:oracle:thin:@//server:1521/SID
> 
> 
> 
> username
> myuser
> 
> 
> 
> password
> mypass
> 
> 
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Are you truly being asked to switch to 2FA?  What is the additional factor?  
Like others have said, supplying something like a code from an RSA token sounds 
exceptionally difficult, however that's not the only possibility.   You 
mentioned a certificate, so I'm wondering whether you're really being asked to 
do mutual authentication, which involves a certificate, but is not as hard as 
actual 2FA.

Also, I assume you have some code that consumes those params from your web.xml. 
 If so, then you might have some flexibility to change the code to do some 
other form of authentication.

John

RE: Why will Tomcat not accept EC cipher suites?

[John.E.Gregg]

Chris,


> -Original Message-
> From: Christopher Schultz [mailto:ch...@christopherschultz.net]
> Sent: Monday, January 08, 2018 8:16 PM
> To: users@tomcat.apache.org
> Subject: Re: Why will Tomcat not accept EC cipher suites?
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> John,
> 
> On 1/8/18 6:28 PM, john.e.gr...@wellsfargo.com.INVALID wrote:
> > Chris and Mark,
> >> -Original Message- From: Christopher Schultz
> >> [mailto:ch...@christopherschultz.net] Sent: Monday, January 08,
> >> 2018 5:21 PM To: users@tomcat.apache.org Subject: Re: Why will Tomcat
> >> not accept EC cipher suites?
> >>
> >> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256
> >>
> >> Mark,
> >>
> >> On 1/8/18 3:36 PM, Mark Thomas wrote:
> >>> On 08/01/18 19:34, john.e.gr...@wellsfargo.com.INVALID wrote:
>  All,
> 
>  I'm using Tomcat 7.0.82 and java 1.8.0_152.
> 
>  I cannot get Tomcat to accept elliptic curve ciphers.  I've written
>  a small SSL socket server that uses the same certificate as the
>  server and deployed it on the same machine using the same JDK.  It
>  accepts EC ciphers just fine so I don't think there is anything in
>  the JDK that has disabled them, etc.  With verbose SSL enabled,
>  Tomcat, however, complains about "http-bio-7114-exec-4, handling
>  exception:
>  javax.net.ssl.SSLHandshakeException: no cipher suites in common."
> 
>  If I omit the "ciphers" property of the connector, I get
>  this:
> 
>  No available cipher suite for TLSv1 No available cipher suite for
>  TLSv1.1 No available cipher suite for TLSv1.2
> 
>  If I set ciphers="ALL,"  I'm back to "no cipher suites in common."
> 
>  If I explicitly tell Tomcat to accept
>  TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, which works with my
> >> socket
>  server, I get "No appropriate protocol (protocol is disabled or
>  cipher suites are inappropriate)."
> 
>  BTW I have an RSA cert on the server with a 2048-bit key and signed
>  using SHA256withRSA.
> 
>  One of the connector configs I've tried.
> 
>    maxThreads="400" maxKeepAliveRequests="100"
>  keepAliveTimeout="1" scheme="https" secure="true"
>  clientAuth="true" sessionCacheSize="5" sslProtocol="TLS"
>  keystoreFile="/path/to/keystore"
>  keystorePass="${keystore.password}" keyAlias="test"
>  truststoreFile="/path/to/cacerts"
>  truststorePass="${truststore.password}"
>  allowUnsafeLegacyRenegotiation="false" />
> >>>
> >>> Try getting it to work without client authentication to start with.
> >>
> >> +1
> >>
> >>> I don't see anything that jumps out as wrong in the above.
> >>
> >> Also, John, what client are you using to test?
> >>
> >> - -chris
> >
> > At Mark's suggestion, I disabled client auth, but it didn't make any
> > difference.  The handshake fails before it even gets to that step.
> >
> > I'm using several different clients, including HP Performance Center,
> > openssl, and a couple of java clients that I wrote myself (one uses
> > SSLSocket directly and one uses HttpsUrlConnection.)
> >
> > Currently I'm looking at the JDK's ServerHandshaker class to make sure
> > I understand the log messages.
> 
> Are you doing something mundane such as:
> 
> $ openssl s_client -connect example.com:8443 ?
> 
> I would expect that to be able to negotiate a TLS connection with a pretty
> standard Tomcat with TLS enabled (and nothing in particular specified for
> ciphers, protocols, etc.).
> 
> - -chris

It turns out that we have elliptic curve ciphers explicitly disabled with the 
system property -Dcom.sun.net.ssl.enableECC=false.  I know the OWASP cheat 
sheet says to favor DHE over ECDHE but I'll have to ask around to find out if 
that's the reason.

Thanks

RE: Why will Tomcat not accept EC cipher suites?

[John.E.Gregg]

Chris and Mark,


> -Original Message-
> From: Christopher Schultz [mailto:ch...@christopherschultz.net]
> Sent: Monday, January 08, 2018 5:21 PM
> To: users@tomcat.apache.org
> Subject: Re: Why will Tomcat not accept EC cipher suites?
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> Mark,
> 
> On 1/8/18 3:36 PM, Mark Thomas wrote:
> > On 08/01/18 19:34, john.e.gr...@wellsfargo.com.INVALID wrote:
> >> All,
> >>
> >> I'm using Tomcat 7.0.82 and java 1.8.0_152.
> >>
> >> I cannot get Tomcat to accept elliptic curve ciphers.  I've written a
> >> small SSL socket server that uses the same certificate as the server
> >> and deployed it on the same machine using the same JDK.  It accepts
> >> EC ciphers just fine so I don't think there is anything in the JDK
> >> that has disabled them, etc.  With verbose SSL enabled, Tomcat,
> >> however, complains about "http-bio-7114-exec-4, handling exception:
> >> javax.net.ssl.SSLHandshakeException: no cipher suites in common."
> >>
> >> If I omit the "ciphers" property of the connector, I get this:
> >>
> >> No available cipher suite for TLSv1 No available cipher suite for
> >> TLSv1.1 No available cipher suite for TLSv1.2
> >>
> >> If I set ciphers="ALL,"  I'm back to "no cipher suites in common."
> >>
> >> If I explicitly tell Tomcat to accept
> >> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, which works with my
> socket
> >> server, I get "No appropriate protocol (protocol is disabled or
> >> cipher suites are inappropriate)."
> >>
> >> BTW I have an RSA cert on the server with a 2048-bit key and signed
> >> using SHA256withRSA.
> >>
> >> One of the connector configs I've tried.
> >>
> >>  >> maxThreads="400" maxKeepAliveRequests="100"
> >> keepAliveTimeout="1" scheme="https" secure="true"
> >> clientAuth="true" sessionCacheSize="5" sslProtocol="TLS"
> >> keystoreFile="/path/to/keystore"
> >> keystorePass="${keystore.password}" keyAlias="test"
> >> truststoreFile="/path/to/cacerts"
> >> truststorePass="${truststore.password}"
> >> allowUnsafeLegacyRenegotiation="false" />
> >
> > Try getting it to work without client authentication to start with.
> 
> +1
> 
> > I don't see anything that jumps out as wrong in the above.
> 
> Also, John, what client are you using to test?
> 
> - -chris

At Mark's suggestion, I disabled client auth, but it didn't make any 
difference.  The handshake fails before it even gets to that step.

I'm using several different clients, including HP Performance Center, openssl, 
and a couple of java clients that I wrote myself (one uses SSLSocket directly 
and one uses HttpsUrlConnection.)

Currently I'm looking at the JDK's ServerHandshaker class to make sure I 
understand the log messages.

Thanks

Why will Tomcat not accept EC cipher suites?

[John.E.Gregg]

All,

I'm using Tomcat 7.0.82 and java 1.8.0_152.

I cannot get Tomcat to accept elliptic curve ciphers.  I've written a small SSL 
socket server that uses the same certificate as the server and deployed it on 
the same machine using the same JDK.  It accepts EC ciphers just fine so I 
don't think there is anything in the JDK that has disabled them, etc.  With 
verbose SSL enabled, Tomcat, however, complains about "http-bio-7114-exec-4, 
handling exception: javax.net.ssl.SSLHandshakeException: no cipher suites in 
common."

If I omit the "ciphers" property of the connector, I get this:

No available cipher suite for TLSv1
No available cipher suite for TLSv1.1
No available cipher suite for TLSv1.2

If I set ciphers="ALL,"  I'm back to "no cipher suites in common."

If I explicitly tell Tomcat to accept TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, 
which works with my socket server, I get "No appropriate protocol (protocol is 
disabled or cipher suites are inappropriate)."

BTW I have an RSA cert on the server with a 2048-bit key and signed using 
SHA256withRSA.

One of the connector configs I've tried.



Thanks

John

Missing 0-length chunk to mark the end of a response

[John.E.Gregg]

All,

Hopefully I can explain this clearly...

The short question is that my client is sometimes not getting the 0-length 
chunk at the end of the response from Tomcat, so my connections don't get 
reused.

My Tomcat server is version 7.0.81.  JDK is 1.8.0_144.  I have a Jersey client, 
version 2.25, connecting to it from another Tomcat server.  I noticed that 
inbound connections to the server are not getting reused properly.  I think the 
reason has to do with the fact that Tomcat is using chunked output but is not 
sending the 0-length chunk that signals the end of the response, even though 
the response is completed/successful.

My Tomcat server has 3 connectors: one plain HTTP, one SSL, and one SSL + 
mutual authentication (2-way SSL.)  With plain HTTP, I always get the 0-length 
chunk.  I verified this both with the remote debugger and TCPMon.  With the 
other two connectors, I don't get the 0-length chunk with the Jersey client, 
though I do see it with some other clients.  I can't use TCPMon with SSL, but 
with the same breakpoints as with non-SSL I see that there is no 0-length chunk.

Setting a breakpoint on ChunkedInputStream line 326 on the client side and 
printing the chunk sizes, I get this output for 2 responses over plain HTTP:

chunk size: 8192
chunk size: 1312
chunk size: 0
chunk size: 8192
chunk size: 1312
chunk size: 0

And for SSL:

chunk size: 8192
chunk size: 1312
chunk size: 8192
chunk size: 1312

When there is no 0-length chunk, the processRaw() method exits on line 304.  
When there is, it exits on line 457.

In all scenarios, the client uses HTTP/1.1 and sends the Connection: keep-alive 
header.  Adding "%{Connection}i %{Connection}o" to my access valve gives me 
this in the log:

keep-alive -

When I run tests from a load generator directly against the server, the 
chunking occurs and the connections get reused as expected.

Here is my plain HTTP connector:



Here is one of my SSL connectors:



Any ideas?

Thanks

John

RE: TLS handshake performance

[John.E.Gregg]

> -Original Message-
> From: Mark Thomas [mailto:ma...@apache.org]
> Sent: Thursday, May 18, 2017 8:47 AM
> To: users@tomcat.apache.org
> Subject: Re: TLS handshake performance
> 
> This time to the right list...
> 
> On 18/05/2017 06:04, Christopher Schultz wrote:
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA256
> >
> > Mark,
> >
> > On 5/17/17 5:31 PM, Mark Thomas wrote:
> >> I got asked in the corridor at TomcatCon earlier today what the
> >> relative performance of the TLS handshake was with 8.5.x, the NIO
> >> connector and JSSE vs OpenSSL TLS implementation.
> > I'm curious about what exactly "TLS handshake" was intended to mean
> > (by the person who asked the question) in this context.
> 
> They are using Tomcat in a scenario where clients are making single requests
> (so keep alve doesn't help). Given that the handshake uses asymmetric
> encryption which is more expensive that symmetric encryption (which is why
> the handshake is used to establish a shared secret so symmetric encryption
> can used for the actual data) they wanted a sense of the performance
> benefit - if any - of NIO and 8.5.x with OpenSSL vs NIO and 8.5.x with JSSE.
> 
> > The handshake itself does not perform any bulk transfer of encrypted
> > data, so the negotiated cipher suite does not matter. However...
> >
> >> Tested with: ab -n 1000 -c 2 -f TLS1.2 -Z
> >> ECDHE-RSA-AES128-GCM-SHA256 https://localhost:8443/test.txt
> >
> > Here the cipher suite matters very much, since the client is not only
> > performing the TLS handshake but also transferring the client's
> > request to the server and the server's response back to the client. >
> > Support for a particular algorithm may dominate the benchmark, here.
> 
> Agreed. But it is the handshake that dominates the timings (if you add -k to
> use keep-alive the req/sec are an order of magnitude higher).
> 
> The cipher suite was the default one chosen by by one of the configs (I
> forget which).
> 
> The cipher suite will affect the results since it also impacts the enctrpyion
> used during the handshake but for any 'reaosnably' secure cipher suite, I'd
> expect similar results in terms of the relative performance.
> 


What about SSL session reuse?  Did you have any way of disabling that?  If 
you're testing against a single server, you'll get 100% reuse, which makes a 
big difference in the CPU load (lower) on the server.  If your real world case 
involves dozens or hundreds of servers behind a load balancer, you get almost 
no reuse.

Thanks

RE: Can't get over 100 client connections?

[John.E.Gregg]

Hi Chris

> -Original Message-
> From: Christopher Schultz [mailto:ch...@christopherschultz.net]
> Sent: Thursday, May 18, 2017 12:06 AM
> To: users@tomcat.apache.org
> Subject: Re: Can't get over 100 client connections?
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> John,
> 
> On 5/16/17 1:57 PM, john.e.gr...@wellsfargo.com.INVALID wrote:
> > All,
> >
> > I'm using Tomcat 7.0.75.
> >
> > I cannot get my connection or thread count over 100 no matter how much
> > load I throw at the server.  Currently I just have a dummy app
> > deployed that doesn't do much except sleep for about 500ms and return
> > a canned response.  Initially I had maxThreads=20 with no explicit
> > maxConnections.  I could get the "current threads busy"
> > metric to 20 easily.  Then I raised it to 100 and was able to reach
> > that number easily.  Additionally, netstat told me I had 100 incoming
> > connections to port 5114.  However raising the maxThreads higher
> > doesn't make any difference, nor does explicitly specifying
> > maxConnections=200.  In either case, I can only get 100 busy threads
> > and 100 active connections.  Once I reach that limit, the response
> > times on the client start going up.  The Tomcat server itself isn't
> > stressed at all.  CPU and GC are low.  The client is on a different
> > server in the same data center.
> >
> > I'm not going through a load balancer.  I'm going directly to the
> > connector below:
> >
> >
> >  > maxConnections="200" maxThreads="200" maxKeepAliveRequests="100"
> > keepAliveTimeout="1" scheme="https" secure="true"
> > clientAuth="true" sslProtocol="TLS"
> > keystoreFile="/app/comp/myapp/certs/appcerttestkeystore"
> > keystorePass="${keystore.password}" keyAlias="test"
> > truststoreFile="/app/comp/myapp/certs/cacerts"
> > truststorePass="${truststore.password}"
> > allowUnsafeLegacyRenegotiation="false" ciphers="blah blah blah" />
> 
> What is the benchmark client and how do you invoke it?
> 
> - -chris

LoadRunner --> vendor app using Commons HttpClient 4.x --> my Tomcat server.  
Tomcat is running an app I wrote to stub out the backend because we're not 
allowed to run load tests against it.  I confirmed that Tomcat itself could 
handle more than 100 connections by running the vendor app under load, as well 
as a test from JMeter directly against my Tomcat server.  Worked fine.  It's 
the vendor app that appears to have hit a 100 connection limit.

Interestingly, JMeter with the option to use HttpClient 4.x opens and closes a 
lot of connections.  The exact same JMeter test with HttpClient 3.x selected 
reuses the connection the way I expect.

If you're on the HttpClient users list, you might see a question from me!  I 
don't understand why HttpClient is behaving the way it is.

Thanks

RE: Can't get over 100 client connections?

[John.E.Gregg]

André,


> -Original Message-
> From: André Warnier (tomcat) [mailto:a...@ice-sa.com]
> Sent: Tuesday, May 16, 2017 2:00 PM
> To: users@tomcat.apache.org
> Subject: Re: Can't get over 100 client connections?
> 
> On 16.05.2017 19:57, john.e.gr...@wellsfargo.com.INVALID wrote:
> > All,
> >
> > I'm using Tomcat 7.0.75.
> >
> > I cannot get my connection or thread count over 100 no matter how much
> load I throw at the server.  Currently I just have a dummy app deployed that
> doesn't do much except sleep for about 500ms and return a canned
> response.  Initially I had maxThreads=20 with no explicit maxConnections.  I
> could get the "current threads busy" metric to 20 easily.  Then I raised it to
> 100 and was able to reach that number easily.  Additionally, netstat told me I
> had 100 incoming connections to port 5114.  However raising the maxThreads
> higher doesn't make any difference, nor does explicitly specifying
> maxConnections=200.  In either case, I can only get 100 busy threads and 100
> active connections.  Once I reach that limit, the response times on the client
> start going up.  The Tomcat server itself isn't stressed at all.  CPU and GC 
> are
> low.  The client is on a different server in the same data center.
> >
> > I'm not going through a load balancer.  I'm going directly to the connector
> below:
> >
> >
> >   >  protocol="HTTP/1.1"
> >  SSLEnabled="true"
> >  maxConnections="200"
> >  maxThreads="200"
> >  maxKeepAliveRequests="100"
> >  keepAliveTimeout="1"
> >  scheme="https"
> >  secure="true"
> >  clientAuth="true"
> >  sslProtocol="TLS"
> >  keystoreFile="/app/comp/myapp/certs/appcerttestkeystore"
> >  keystorePass="${keystore.password}"
> >  keyAlias="test"
> >  truststoreFile="/app/comp/myapp/certs/cacerts"
> >  truststorePass="${truststore.password}"
> >  allowUnsafeLegacyRenegotiation="false"
> >  ciphers="blah blah blah"
> >  />
> >
> 
> Hi.
> I do not know with what you are testing (as a client).
> But be aware of the following :
> 
> 1) >  keepAliveTimeout="1"
> means 10 seconds.
> It means that, after the last request which one particular client sends on its
> connection to Tomcat, and Tomcat has responded to it, Tomcat will keep that
> connection open for an additional 10 s., just waiting to see if that same 
> client
> has anything more to request.
> Since you are not using an Executor, keeping the connection open will also
> mean keeping the corresponding Tomcat thread alive, also waiting.
> Only once this time is over, will Tomcat close this connection, and "recycle"
> the thread to serve another client connection.
> 2) there may be a limit in the server OS, as to how many connections a
> process can have open at the same time. If that limit is reached at some
> point, that may either crash the process that wants an additional one, or put
> it in some wait queue until one is available again.
> 3) when a client opens a connection to a server (or tries to), and the server
> process does not immediately respond to the "open connection" request,
> the TCP/IP stack on the server will place the connection-open request in a
> wait queue. The size of that queue is an adjustable TCP/IP parameter.
>  From the client side, if its connection is not accepted immediately (but not
> rejected right away), the client will just wait, until it is accepted. There 
> is
> usually a timeout for this also, on the client side.
> 
> Some combination of the above may explain what you see.
> 

Thanks for the suggestions.  I don't know 100% the cause yet but it appears to 
be an issue on the client side.  When running multiple clients I can get the 
concurrent requests over 100 with no trouble.

Thanks


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Can't get over 100 client connections?

[John.E.Gregg]

All,

I'm using Tomcat 7.0.75.

I cannot get my connection or thread count over 100 no matter how much load I 
throw at the server.  Currently I just have a dummy app deployed that doesn't 
do much except sleep for about 500ms and return a canned response.  Initially I 
had maxThreads=20 with no explicit maxConnections.  I could get the "current 
threads busy" metric to 20 easily.  Then I raised it to 100 and was able to 
reach that number easily.  Additionally, netstat told me I had 100 incoming 
connections to port 5114.  However raising the maxThreads higher doesn't make 
any difference, nor does explicitly specifying maxConnections=200.  In either 
case, I can only get 100 busy threads and 100 active connections.  Once I reach 
that limit, the response times on the client start going up.  The Tomcat server 
itself isn't stressed at all.  CPU and GC are low.  The client is on a 
different server in the same data center.

I'm not going through a load balancer.  I'm going directly to the connector 
below:




Thanks

John

RE: Increasing maxThreads results in more "Connection refused" errors?

[John.E.Gregg]

Chris

> -Original Message-
> From: Christopher Schultz [mailto:ch...@christopherschultz.net]
> Sent: Friday, February 03, 2017 11:08 AM
> To: Tomcat Users List
> Subject: Re: Increasing maxThreads results in more "Connection refused" 
> errors?
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> John,
> 
> On 2/3/17 11:17 AM, john.e.gr...@wellsfargo.com.INVALID wrote:
> > I'm doing some scalability testing of an app in Tomcat 7.0.73.
> >
> > Some relevant connector config details: maxThreads=80
> > maxKeepAliveRequests=100 keepAliveTimeout=1 maxConnections
> > unspecified (defaults to maxThreads according to the docs) acceptCount
> > unspecified (100 according to the docs) clientAuth=true
> 
> Which connector?
> 
> > FWIW, I'm testing two Tomcat instances on the same server.  They are
> > behind a load balancer.
> 
> Which load-balancer using which protocol?
> 
> > It appears that when the load generator tries to exceed maxThreads,
> > the new connection rate goes up quickly and CPU usage shoots up with
> > it.
> 
> When you say the "new connection rate" goes up, what exactly do you mean?
> When using a load-generator, I would certainly expect the "new connection
> rate" to go up.
> 
> > I assume this is because Tomcat is proactively closing idle keep alive
> > connections to service new connections.
> 
> No. Tomcat behaves the same under load as when not under load. If you are
> using the BIO connector (it sounds like you are), then each connection will 
> have
> to wait for the KeepAlive timeout to expire before servicing another request
> from another client.
> 
> This is complicated by the reverse-proxy and so those details are important to
> provide.
> 
> > In an effort to keep the CPU in check, I tried increasing maxThreads
> > from 80 to 120.
> 
> If the CPU is very busy, allowing more threads to run is counter-productive, 
> isn't
> it?
> 
> > This seemed to work well in a lot of ways. New connection rate didn't
> > increase as much, CPU didn't increase as much, there was more
> > connection reuse (more requests per connection,) and response times
> > didn't deteriorate as much.
> >
> > Great, right?
> >
> > Then I noticed a large increase in "Connection refused" errors on the
> > load generator. In other words, a higher maxThreads also results in a
> > high error rate. The total hits per second from the client's
> > perspective is about 60 in both cases. With maxThreads=80, there are
> > about 3 connection refused errors per second at that volume. With
> > maxThreads=120, there are about 10 connection refused errors per
> > second.
> 
> Are you hitting the load-balancer or Tomcat directly during your load-test?
> 
> If you are hitting the load-balancer, then the details of that configuration 
> are
> more important for the client: if the lb will only accept 10 connections, it 
> doesn't
> matter what Tomcat is willing to accep t.
> 
> > I have no idea why this is.  Can someone explain this or what I can do
> > about it?
> 
> Post more data. We'll get to the bottom of it.
> 
> - -chris

Thanks for asking those good questions.  When I checked with the load balancer 
people, they told me there was a 70 connection limit to each Tomcat server 
enforced by the load balancer.  It took a while to get that limit removed, but 
now that it's gone, the connection errors completely went away.  I think the 
limit is a good idea and we will probably re-establish a higher one after 
additional testing.

Thanks

John

Increasing maxThreads results in more "Connection refused" errors?

[John.E.Gregg]

All,

I'm doing some scalability testing of an app in Tomcat 7.0.73.

Some relevant connector config details:
maxThreads=80
maxKeepAliveRequests=100
keepAliveTimeout=1
maxConnections unspecified (defaults to maxThreads according to the docs)
acceptCount unspecified (100 according to the docs)
clientAuth=true

FWIW, I'm testing two Tomcat instances on the same server.  They are behind a 
load balancer.

It appears that when the load generator tries to exceed maxThreads, the new 
connection rate goes up quickly and CPU usage shoots up with it.  I assume this 
is because Tomcat is proactively closing idle keep alive connections to service 
new connections.  In an effort to keep the CPU in check, I tried increasing 
maxThreads from 80 to 120.  This seemed to work well in a lot of ways.  New 
connection rate didn't increase as much, CPU didn't increase as much, there was 
more connection reuse (more requests per connection,) and response times didn't 
deteriorate as much.

Great, right?

Then I noticed a large increase in "Connection refused" errors on the load 
generator.  In other words, a higher maxThreads also results in a high error 
rate.  The total hits per second from the client's perspective is about 60 in 
both cases.  With maxThreads=80, there are about 3 connection refused errors 
per second at that volume.  With maxThreads=120, there are about 10 connection 
refused errors per second.

I have no idea why this is.  Can someone explain this or what I can do about it?

Thanks

John

RE: Please help with Tomcat Garbage Collection

[John.E.Gregg]

> -Original Message-
> From: George I. Develekos [mailto:gdevele...@omilia.com]
> Sent: Wednesday, November 16, 2016 10:18 AM
> To: users@tomcat.apache.org
> Subject: Re: Please help with Tomcat Garbage Collection
> 
> I appreciate the detailed response.
> 
> On another installation with higher  load, the JVM has "selected" to give
> YoungGen 250MB or so (as opposed to 150M here), and I have confirmed that
> Full-GC is much less frequent so I'll go with that next time I have downtime.
> 
> When exactly can I expect to see my app freeze? Isn't it during Full-GC?
> 
> Does the increase in CPU during those Full-GC times make sense?
> 
> 
> 
> Regards,
> 
> George I. Develekos | SeniorSW Engineer |t: _+30.210.6930664_| e:
> gdevele...@omilia.com 
> 
> 
> 
> Technology that Listens, Understands and Cares   - www.omilia.com
> 
> 
> On 11/16/2016 6:11 PM, john.e.gr...@wellsfargo.com wrote:
> >
> > Sorry for top posting.  The format got weird.
> >
> > Those numbers aren’t bad.  The most important number for me is the
> > throughput on the summary tab.  Yours is 99.38%.  That means the JVM
> > was doing real work 99.38% of the time and garbage collecting the
> > other .62%.  You could improve that if you worked hard enough, but
> > going from 99.4 to 99.5 or 99.6 probably isn’t worth the effort.  If
> > the number was 90% or something, then you’d have more room for
> > improvement.
> >
> > Look at the GC performance numbers on the summary tab.  Obviously
> > minor GCs are much faster in this regard than major GCs. You can
> > reduce your total GC time by increasing the size of your young
> > generation.  You will get more or slower young collections but fewer
> > and faster old collections.  Overall the total time will be less than
> > it is now and the longest pauses will be shorter.
> >
> > As others have said, though, something doesn’t add up.  CMS is only
> > stop-the-world during certain phases.  (Not the ones with “concurrent”
> > in the name.)  If you feel these GC events coincide with pauses in
> > your app, you can try a thread dump or three (kill -3 ) during
> > the pause.  Use a tool like Samurai to parse the output.  This might
> > only be practical for longer pauses, though.
> >
> > Also, is it possible the VM itself is having a problem?  Maybe you
> > should talk to your virtualization team to see how stressed the
> > hardware is.  VMWare has an informative java best practices doc:
> >
> http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techp
> > aper/enterprise-java-applications-on-vmware-best-practices-guide.pdf
> >
> > John
> >
> > *From:*George I. Develekos [mailto:gdevele...@omilia.com]
> > *Sent:* Wednesday, November 16, 2016 5:21 AM
> > *To:* users@tomcat.apache.org
> > *Subject:* Re: Please help with Tomcat Garbage Collection
> >
> > I'm attaching three screenshots of the GCViewer app as it processed
> > the complete *gc.log* file (about 19 hours).
> >
> > Please have a look and advise on what I can do to limit Full-GC times.
> > As of now I have a recommendation to increase the Young Gen..
> >
> > The setup in summary:
> >
> > We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
> > 7.0.64.
> >
> > Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
> > -XX:+CMSIncrementalMode
> >
> >
> >
> >
> >
> >
> > On 11/15/2016 11:45 PM, john.e.gr...@wellsfargo.com
> >  wrote:
> >
> > -Original Message-
> >
> > From: George I. Develekos [mailto:gdevele...@omilia.com]
> >
> > Sent: Tuesday, November 15, 2016 3:00 PM
> >
> > To:users@tomcat.apache.org 
> >
> > Subject: Re: Please help with Tomcat Garbage Collection
> >
> > The system does very little swapping, both when it's GC'ing and 
> > when it's
> not.
> >
> > Less than 100MB worth of swap is taken.
> >
> > Giving Tomcat its own HW is not an option at this time,
> > especially as there's no
> >
> > guarantee it'll solve the problem. Besides it would be a VM
> > anyway, not physical
> >
> > dedicated HW.  The current server is also a VM.
> >
> > On 15-Nov-16 10:55 PM, Zdeněk Henek wrote:
> >
> > I would start with moving this tomcat to its own hw.
> >
> > Did you check swap? This long pauses could be because part
> > of your
> >
> > heap is swapped to hdd
> >
> > Regards,
> >
> > Zdenek Henek
> >
> > On Tue, Nov 15, 2016, 21:37 George I. Develekos
> >
> >  
> >
> > wrote:
> >
> > On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
> >
> > George,
> >
> > On 11/15/16 10:46 AM, George I. Develekos wrote:
> >
> > He

RE: Please help with Tomcat Garbage Collection

[John.E.Gregg]

Sorry for top posting.  The format got weird.

Those numbers aren’t bad.  The most important number for me is the throughput 
on the summary tab.  Yours is 99.38%.  That means the JVM was doing real work 
99.38% of the time and garbage collecting the other .62%.  You could improve 
that if you worked hard enough, but going from 99.4 to 99.5 or 99.6 probably 
isn’t worth the effort.  If the number was 90% or something, then you’d have 
more room for improvement.

Look at the GC performance numbers on the summary tab.  Obviously minor GCs are 
much faster in this regard than major GCs.  You can reduce your total GC time 
by increasing the size of your young generation.  You will get more or slower 
young collections but fewer and faster old collections.  Overall the total time 
will be less than it is now and the longest pauses will be shorter.

As others have said, though, something doesn’t add up.  CMS is only 
stop-the-world during certain phases.  (Not the ones with “concurrent” in the 
name.)  If you feel these GC events coincide with pauses in your app, you can 
try a thread dump or three (kill -3 ) during the pause.  Use a tool like 
Samurai to parse the output.  This might only be practical for longer pauses, 
though.

Also, is it possible the VM itself is having a problem?  Maybe you should talk 
to your virtualization team to see how stressed the hardware is.  VMWare has an 
informative java best practices doc: 
http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/enterprise-java-applications-on-vmware-best-practices-guide.pdf

John




From: George I. Develekos [mailto:gdevele...@omilia.com]
Sent: Wednesday, November 16, 2016 5:21 AM
To: users@tomcat.apache.org
Subject: Re: Please help with Tomcat Garbage Collection

I'm attaching three screenshots of the GCViewer app as it processed the 
complete gc.log file (about 19 hours).

Please have a look and advise on what I can do to limit Full-GC times. As of 
now I have a recommendation to increase the Young Gen..

The setup in summary:

We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC   
-XX:+CMSIncrementalMode


[cid:image001.png@01D23FEE.84B81AB0]

[cid:image002.png@01D23FEE.84B81AB0]


[cid:image003.png@01D23FEE.84B81AB0]
On 11/15/2016 11:45 PM, 
john.e.gr...@wellsfargo.com wrote:







-Original Message-

From: George I. Develekos [mailto:gdevele...@omilia.com]

Sent: Tuesday, November 15, 2016 3:00 PM

To: users@tomcat.apache.org

Subject: Re: Please help with Tomcat Garbage Collection



The system does very little swapping, both when it's GC'ing and when it's not.

Less than 100MB worth of swap is taken.



Giving Tomcat its own HW is not an option at this time, especially as there's no

guarantee it'll solve the problem. Besides it would be a VM anyway, not physical

dedicated HW.  The current server is also a VM.





On 15-Nov-16 10:55 PM, Zdeněk Henek wrote:

I would start with moving this tomcat to its own hw.



Did you check swap? This long pauses could be because part of your

heap is swapped to hdd



Regards,

Zdenek Henek



On Tue, Nov 15, 2016, 21:37 George I. Develekos



wrote:



On 15-Nov-16 10:22 PM, Christopher Schultz wrote:

George,



On 11/15/16 10:46 AM, George I. Develekos wrote:

Hello guys,



We are having problems on a production system with very long "full

GC" times, as long as1200sec real time (!!!).



We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat

7.0.64.



Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC

-XX:+CMSIncrementalMode



No other custom memory-related settings are in place.



Looking at the GC log, the last few Full-GC entries are:



1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),

2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :

169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:

user=1.96 sys=0.13, real=2.11 secs]



2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:

2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,

real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:

[CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:

user=6.12 sys=1.36, real=*23.21 secs*]



4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),

2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :

169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:

user=2.18 sys=0.14, real=2.34 secs]





What can we do?

1367.020 Full GC duration=2.11s

2579.317 Full GC duration=1212.49s



So your full GC immediately started another full GC that took 20

minutes ?



Are you only showing certain FULL GC activity from your log, or is

that everything?



CMS should have a mark and then a sweep each time, but your times

don't seem to add up.



also note that the whole point of CMS is that

RE: Problems with SSL configuration

[John.E.Gregg]

Enable verbose SSL.

Start Tomcat with -Djavax.net.debug=ssl. That will print a lot of info to 
catalina.out.

You could also do the same thing on the client side if you used a java client, 
or something similar with OpenSSL, curl, etc.



-Original Message-
From: Steve Willett 
[st...@yoursportsleague.com]
Sent: Tuesday, November 15, 2016 05:48 PM Central Standard Time
To: users@tomcat.apache.org
Subject: Problems with SSL configuration


I am trying to set up a stand-alone Tomcat server (apparently 7.0.53).
When I set up a simple Connector on port 8443 (no specified ciphers, and
a simple sslProtocol="TLS") using a DigiCert Certificate I can connect.

However, if I test it with QualSys, I get an F rating because of the
accepted insecure cipher suites.  However, when I try to use "approved"
suites, the server can't be reached.

Connector configuration;
 

When I try to connect to the site with Chrome I get:


  This site can’t be reached

*qa.yoursportsleague.com*unexpectedly closed the connection.



I also have configured it to require SSL:




Protected Context
/*



CONFIDENTIAL





Any thoughts?


--
*Steve Willett*
YourSportsLeague.com

RE: 8.5.4 to 8.5.5 SSL Issue

[John.E.Gregg]

> -Original Message-
> From: William Boyd [mailto:william.b...@gmail.com]
> Sent: Tuesday, November 15, 2016 3:44 PM
> To: Tomcat Users List
> Subject: Re: 8.5.4 to 8.5.5 SSL Issue
> 
> On Tue, Nov 15, 2016 at 10:50 AM, Christopher Schultz <
> ch...@christopherschultz.net> wrote:
> 
> > -BEGIN PGP SIGNED MESSAGE-
> > Hash: SHA256
> >
> > William,
> >
> > On 11/14/16 7:18 PM, William Boyd wrote:
> > > First, I'd like to thank everyone for the help.
> > >
> > > Is it now safe to say that the behaviour we've been taking advantage
> > > of is undocumented and will no long be supported?
> > >
> > > Also, for those that hit this thread and need to get HTTPS working
> > > with a *self-signed* certificate in a dev environment...
> > >
> > > Here's what worked for me: 1. Copy
> > > %JAVA_HOME%\jre\lib\security\cacerts some place (say C:\keystore) 2.
> > > Create a *self-signed* certificate with %JAVA_HOME%\bin\keytool
> > > -genkeypair -keyalg RSA -alias myAlias -keystore
> > > "C:\keystore\keystore.jsk" -storepass changeit -validity 360
> > > -keysize 2048 -dname
> > > CN=localhost,OU=OrgUnit,O=Org,L=City,ST=State,C=Country 3. Export
> > > the myAlias certificate with %JAVA_HOME%\bin\keytool -export -alias
> > > myAlias -keystore C:\keystore\keystore.jsk -rfc -file
> > > C:\keystore\myAlias.cer 4. Import the myAlias certificate into your
> > > copy of cacerts with: %JAVA_HOME%\bin\keytool -import -alias myAlias
> > > -keystore C:\keystore\cacerts -file C:\keystore\myAlias.cer 5. Add
> > > this to setclasspath.bat in tomcat:
> > > set JAVA_OPTS=%JAVA_OPTS%
> > > -Djavax.net.ssl.trustStore="C:\keystore\cacert" 6. In server.xml,
> > > add these attributes to the Connector element
> > > keystoreFile="C:\keystore\keystore.jsk" keystorePass="changeit"
> > > keyAlias="myAlias" scheme="https" secure="true" SSLEnabled="true"
> > > clientAuth="false" sslProtocol="TLS"
> > >
> > > Now you should be good to go in with HTTPS in Tomcat 8.5.5 +
> >
> > This is exactly what I suggested, except that you set the trust store
> > using a system property instead of using truststoreFile in the
> > . You do not need to duplicate the JRE's trust store. You
> > only need your own single certificate in your local truststore.
> >
> > Can you use keystoreFile and truststoreFile separately pointing to
> > those files and re-check that it works? Perhaps Tomcat chokes when
> > using the same file for both. That's what I'd like to verify.
> >
> > - -chris
> > -BEGIN PGP SIGNATURE-
> > Comment: GPGTools - http://gpgtools.org
> > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> >
> >
> iQIcBAEBCAAGBQJYK1kHAAoJEBzwKT+lPKRYa48P/jl6hWa1mj5HCnawJZ3mHbjX
> >
> ADdXYl45aym/E6QV/n99XYVYG2q+ZN9w0XDVP54lQhQhcgOUtCiPbTHEcYSYdwr
> U
> >
> kLYMc3Ge8Jt7/zDMvem+pKYkHMvyHbspVqSujZ4uJ3Ozr9mYD89hSFgxqG0iYSE+
> > 5c0pvz1nW4Pt1F4A/+WETkL4Y5Xrq1Vn1LSAxAZoYiU/o93nVos7etIBUO9E430+
> > GihbhvkpS/yBitvrir/YacvWauBxpi30wR++6ZNAhpzlb+j90dk3i6iPcDO6K1f2
> > SNeqZATJDlXyU1hEksW4UxWLhtUeekqmJEiEqqWCYxNz9lwJG9f4kILUrzsZexlu
> >
> FmP2o4IxWTBcgOUs5Km5DlfYwogJmlRhqQoOlg2JOpv+KIb67DX+PuY6bhGomDf
> f
> >
> YQ03Y7WQcjNZ/uOIoadAkXxKRaRHmuz2KkPYwgDutOgxtJV1jNxTT3A3znGT1cW
> N
> >
> yekjXHOpe2FdXnaoG0X7mTpvx5AhkHN9mRdW+5/ZBpPzUN0M7zy8oBEpLtZKfrT
> J
> > k40Xz70DnNxBP3XS/1w7DJ1H3/FBxNdatVVbbcJ/+lS/NiS4Gn2kMAZgrCuZrUsn
> >
> FdpdyCwq3VLJ2X9LVBR03rJOyPIiybANNjfhPpiEMC9uQu2ENm4A4Hm1p/cXdpo3
> > 2J2O1AQA7tfew10t3F4K
> > =a+Um
> > -END PGP SIGNATURE-
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
> Hi Christopher,
> 
> Thanks, I'm glad to hear I'm heading in the right direction. :)
> 
> I tested some more configuration.
> 
> Firstly, I removed the system property but retained these Connector attributes
> 
>keystoreFile="C:\keystore\keystore.jsk" keystorePass="changeit"
> keyAlias="myAlias"
>scheme="https" secure="true" SSLEnabled="true" clientAuth="false"
> sslProtocol="TLS"
> 
> Then I tried the following:
> 
> 1. adding truststoreFile="C:\keystore\cacerts" to my Connector and got the
> following error in an IE11 browser
>Caught Exception (javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException:
>PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException:
>unable to find valid certification path to requested target): ;
>nested exception is: javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException:
>PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException:
>unable to find valid certification path to requested target
> 
> 2. adding truststoreFile="C:\keystore\myAlias.cer" to my Connector and got the
> following error at startup
>15-Nov-2016 12:34:57.379 SEVERE [main]
> org.apache.coyote.AbstractProtocol.init
>Failed to initializ

RE: Please help with Tomcat Garbage Collection

[John.E.Gregg]

> -Original Message-
> From: George I. Develekos [mailto:gdevele...@omilia.com]
> Sent: Tuesday, November 15, 2016 3:00 PM
> To: users@tomcat.apache.org
> Subject: Re: Please help with Tomcat Garbage Collection
> 
> The system does very little swapping, both when it's GC'ing and when it's not.
> Less than 100MB worth of swap is taken.
> 
> Giving Tomcat its own HW is not an option at this time, especially as there's 
> no
> guarantee it'll solve the problem. Besides it would be a VM anyway, not 
> physical
> dedicated HW.  The current server is also a VM.
> 
> 
> On 15-Nov-16 10:55 PM, Zdeněk Henek wrote:
> > I would start with moving this tomcat to its own hw.
> >
> > Did you check swap? This long pauses could be because part of your
> > heap is swapped to hdd
> >
> > Regards,
> > Zdenek Henek
> >
> > On Tue, Nov 15, 2016, 21:37 George I. Develekos
> > 
> > wrote:
> >
> >> On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
> >>> George,
> >>>
> >>> On 11/15/16 10:46 AM, George I. Develekos wrote:
>  Hello guys,
> 
>  We are having problems on a production system with very long "full
>  GC" times, as long as1200sec real time (!!!).
> 
>  We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
>  7.0.64.
> 
>  Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
>  -XX:+CMSIncrementalMode
> 
>  No other custom memory-related settings are in place.
> 
>  Looking at the GC log, the last few Full-GC entries are:
> 
>  1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
>  2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
>  169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
>  user=1.96 sys=0.13, real=2.11 secs]
> 
>  2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
>  2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
>  real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
>  [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
>  user=6.12 sys=1.36, real=*23.21 secs*]
> 
>  4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
>  2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
>  169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
>  user=2.18 sys=0.14, real=2.34 secs]
> 
> 
>  What can we do?
> >>> 1367.020 Full GC duration=2.11s
> >>> 2579.317 Full GC duration=1212.49s
> >>>
> >>> So your full GC immediately started another full GC that took 20
> >>> minutes ?
> >>>
> >>> Are you only showing certain FULL GC activity from your log, or is
> >>> that everything?
> >>>
> >>> CMS should have a mark and then a sweep each time, but your times
> >>> don't seem to add up.
> >>>
> >>> also note that the whole point of CMS is that there isn't any
> >>> stop-the-world during the mark portion of the process.
> >>>
> >>> Are you actually experiencing a problem, or are you just suffering
> >>> from instrumentor's remorse?
> >>>
> >>> - -chris
> >>>
> >> Chris,
> >>
> >> What I listed is the result of the command:
> >>
> >> grep "Full GC" gc.log
> >>
> >> So (obviously) I have skipped other GC activity, i.e. whatever GC
> >> activity didn't include the "Full GC" string.
> >>
> >> Yes we are having app trouble due to the GC delays so this is a real
> >> problem. Our application has real-time constraints so the GC delays
> >> cannot be tolerated. I selected those GC options _in order to avoid
> >> _long GC times.
> >>
> >> Additionally, these periods coincide with high CPU for that JVM
> >> process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
> >> Once GC is done, our app rushes to catch up with tasks that had to
> >> wait for GC to finish.
> >>
> >> Answering another question from a member who has kindly responded,
> >> yes the server is running other stuff. Basically it runs three
> >> tomcats, the main one being this one. It also runs a DB2 database
> >> that has close-to-zero activity.
> >>
> >> George
> >>
> >>

It might be helpful if you could post a larger chunk of your GC log, at least 
long enough to cover the start and end of the CMS phases and maybe even more.  
Additionally, try using a tool like GCViewer to analyze the log.

How many CPUs do you have?  60% CPU usage isn't usually a big deal.

Like Chris already said, this is not a stop-the-world phase, so your 
application should continue working in parallel with the garbage collector.

Looks like your young generation is only 150MB (2080128k - 1926784k.)  That's 
very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is the 
JVM choosing that for you?  It's so small that your old generation might be 
filling up faster that it should, leading to more frequent full collections.  
You could try setting the young generation to something like 25-50% of the 
total heap.  You'd get a lot of small pauses as the young gen is collected but 
fewer long ones.

John

RE: Tomcat - Two Way SSL as Server

[John.E.Gregg]

> -Original Message-
> From: Robert Sulliman [mailto:robert.sulli...@sjrb.ca]
> Sent: Monday, November 14, 2016 2:46 PM
> To: Tomcat Users List
> Subject: RE: Tomcat - Two Way SSL as Server
> 
> Thanks John,
> 
> I am trying to do #2, manually adding client certificates to the trust store.
> However it doesn't work unless I add the root certificate to the trust store 
> as
> well, or I get the certificate chain error below. It is a headache to handle 
> certs
> like this, but as a rule of thumb we leave the responsibility for these certs 
> on the
> client themselves.
> 
> I'm pretty sure I'm not going to persuade security to create a new CA for me 
> just
> for this one service... If I use a custom servlet, I lose the ability to do 
> revocation
> checks on the certificates (I'm assuming that Tomcat does this natively, I 
> haven't
> actually tested it yet.)
> 
> Robert Sulliman
> 
> -Original Message-
> From: john.e.gr...@wellsfargo.com [mailto:john.e.gr...@wellsfargo.com]
> Sent: Monday, November 14, 2016 1:24 PM
> To: users@tomcat.apache.org
> Subject: RE: Tomcat - Two Way SSL as Server
> 
> 
> 
> 
> 
> > -Original Message-
> > From: Robert Sulliman [mailto:robert.sulli...@sjrb.ca]
> > Sent: Monday, November 14, 2016 12:25 PM
> > To: users@tomcat.apache.org
> > Subject: Tomcat - Two Way SSL as Server
> >
> > Hi All,
> >
> > I'm trying to implement two way SSL on a new web service that we are
> > building and I'm having some issues.
> >
> > First some info on  the environment.
> >
> > Server version: Apache Tomcat/8.0.36
> > Server built:   Jun 9 2016 13:55:50 UTC
> > Server number:  8.0.36.0
> > OS Name:Linux
> > OS Version: 3.10.0-514.el7.x86_64
> > Architecture:   amd64
> > JVM Version:1.8.0_111-b14
> > JVM Vendor: Oracle Corporation
> >
> > We use an internal certificate authority to sign all of our
> > certificates. So all the client certificates are signed by our
> > internal root. When I trust the root certificate in the client trust
> > store everything works. All client certificates signed by the internal root 
> > work.
> >
> > However, if I remove the root certificate from the client trust store,
> > and add individual client certificates instead I get a cert chain error.
> > 
> > *** ECDH ServerKeyExchange
> > Signature Algorithm SHA512withRSA
> > Server key: Sun EC public key, 256 bits
> >   public x coord:
> >
> 10710875017633521043383492698333011680577506891922716697438973534
> > 1685270962458
> >   public y coord:
> >
> 9319572573423690274300646937808706820914905809794852649056260
> > 79337507
> >   parameters: secp256r1 [NIST P-256, X9.62 prime256v1]
> > (1.2.840.10045.3.1.7)
> > *** CertificateRequest
> > Cert Types: RSA, DSS, ECDSA
> > Supported Signature Algorithms: SHA512withECDSA, SHA512withRSA,
> > SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA,
> > SHA256withDSA, SHA224withECDSA, SHA224withRSA, SHA224withDSA,
> > SHA1withECDSA, SHA1withRSA, SHA1withDSA Cert Authorities:
> >  > ST=Alberta, C=CA>
> > *** ServerHelloDone
> > http-nio2-8443-exec-4, WRITE: TLSv1.2 Handshake, length = 4482
> > http-nio2- 8443-exec-2, READ: TLSv1.2 Handshake, length = 7
> > *** Certificate chain
> > 
> > ***
> > http-nio2-8443-exec-2, fatal error: 42: null cert chain
> > javax.net.ssl.SSLHandshakeException: null cert chain %% Invalidated:
> > [Session- 2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
> > http-nio2-8443-exec-2, SEND TLSv1.2 ALERT:  fatal, description =
> > bad_certificate http-nio2-8443-exec-2, WRITE: TLSv1.2 Alert, length =
> > 2 http- nio2-8443-exec-2, fatal: engine already closed.  Rethrowing
> > javax.net.ssl.SSLHandshakeException: null cert chain
> > http-nio2-8443-exec-2, called closeOutbound() http-nio2-8443-exec-2,
> > closeOutboundInternal()  This is an
> > issue for us as we can't have all the client certificates in the
> > company granted access to this endpoint, it kind of defeats the purpose.
> >
> > The company root certificate is in another trust store used on server 
> > startup.
> > Here are my configs.
> >
> > Server.xml connector:
> > 
> > >port="8443" maxThreads="24" minSpareThreads="4"
> > maxSpareThreads="4" acceptCount="1000" server=" "
> >scheme="https" secure="true" SSLEnabled="true"
> >keystoreFile="certs/servercert.jks" keystorePass="
> CrazyPasswordHere"
> >clientAuth="true"
> > truststoreFile="/usr/local/tomcat/certs/clienttrust.jks"
> > truststorePass="CrazyPasswordHere"
> >sslEnabledProtocols="TLSv1.2" sslProtocol="TLS"
> >
> >
> ciphers="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WIT
> > H_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
> >
> >
> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_128_C
> > BC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA"
> >useServerCipherSuitesOrder="tr

RE: Tomcat - Two Way SSL as Server

[John.E.Gregg]

> -Original Message-
> From: Robert Sulliman [mailto:robert.sulli...@sjrb.ca]
> Sent: Monday, November 14, 2016 12:25 PM
> To: users@tomcat.apache.org
> Subject: Tomcat - Two Way SSL as Server
> 
> Hi All,
> 
> I'm trying to implement two way SSL on a new web service that we are building
> and I'm having some issues.
> 
> First some info on  the environment.
> 
> Server version: Apache Tomcat/8.0.36
> Server built:   Jun 9 2016 13:55:50 UTC
> Server number:  8.0.36.0
> OS Name:Linux
> OS Version: 3.10.0-514.el7.x86_64
> Architecture:   amd64
> JVM Version:1.8.0_111-b14
> JVM Vendor: Oracle Corporation
> 
> We use an internal certificate authority to sign all of our certificates. So 
> all the
> client certificates are signed by our internal root. When I trust the root
> certificate in the client trust store everything works. All client 
> certificates signed
> by the internal root work.
> 
> However, if I remove the root certificate from the client trust store, and add
> individual client certificates instead I get a cert chain error.
> 
> *** ECDH ServerKeyExchange
> Signature Algorithm SHA512withRSA
> Server key: Sun EC public key, 256 bits
>   public x coord:
> 10710875017633521043383492698333011680577506891922716697438973534
> 1685270962458
>   public y coord:
> 9319572573423690274300646937808706820914905809794852649056260
> 79337507
>   parameters: secp256r1 [NIST P-256, X9.62 prime256v1] (1.2.840.10045.3.1.7)
> *** CertificateRequest
> Cert Types: RSA, DSS, ECDSA
> Supported Signature Algorithms: SHA512withECDSA, SHA512withRSA,
> SHA384withECDSA, SHA384withRSA, SHA256withECDSA, SHA256withRSA,
> SHA256withDSA, SHA224withECDSA, SHA224withRSA, SHA224withDSA,
> SHA1withECDSA, SHA1withRSA, SHA1withDSA Cert Authorities:
>  C=CA>
> *** ServerHelloDone
> http-nio2-8443-exec-4, WRITE: TLSv1.2 Handshake, length = 4482 http-nio2-
> 8443-exec-2, READ: TLSv1.2 Handshake, length = 7
> *** Certificate chain
> 
> ***
> http-nio2-8443-exec-2, fatal error: 42: null cert chain
> javax.net.ssl.SSLHandshakeException: null cert chain %% Invalidated:  
> [Session-
> 2, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256]
> http-nio2-8443-exec-2, SEND TLSv1.2 ALERT:  fatal, description =
> bad_certificate http-nio2-8443-exec-2, WRITE: TLSv1.2 Alert, length = 2 http-
> nio2-8443-exec-2, fatal: engine already closed.  Rethrowing
> javax.net.ssl.SSLHandshakeException: null cert chain http-nio2-8443-exec-2,
> called closeOutbound() http-nio2-8443-exec-2, closeOutboundInternal()
>  This is an issue for us as we can't have
> all the client certificates in the company granted access to this endpoint, 
> it kind
> of defeats the purpose.
> 
> The company root certificate is in another trust store used on server startup.
> Here are my configs.
> 
> Server.xml connector:
> 
>port="8443" maxThreads="24" minSpareThreads="4"
> maxSpareThreads="4" acceptCount="1000" server=" "
>scheme="https" secure="true" SSLEnabled="true"
>keystoreFile="certs/servercert.jks" keystorePass=" 
> CrazyPasswordHere"
>clientAuth="true"
> truststoreFile="/usr/local/tomcat/certs/clienttrust.jks"
> truststorePass="CrazyPasswordHere"
>sslEnabledProtocols="TLSv1.2" sslProtocol="TLS"
> 
> ciphers="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WIT
> H_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
> 
> TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_128_C
> BC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA"
>useServerCipherSuitesOrder="true" compression="on"
> compressionMinSize="2048"
> 
> compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,
> application/javascript" />  Systemd init:
> 
> # Systemd unit file for tomcat
> [Unit]
> Description=Apache Tomcat
> After=syslog.target network.target
> 
> [Service]
> Type=forking
> 
> Environment=JAVA_HOME=/usr/lib/jvm/jre
> Environment=CATALINA_PID=/usr/local/tomcat/temp/tomcat.pid
> Environment=CATALINA_HOME=/usr/local/tomcat
> Environment=CATALINA_BASE=/usr/local/tomcat
> Environment='CATALINA_OPTS= -Xms2048M -Xmx2048M -server -
> XX:+UseParallelGC \ -Dcom.sun.management.jmxremote \
> -Dcom.sun.management.jmxremote.port=8090 \ -
> Dcom.sun.management.jmxremote.ssl=false \ -
> Dcom.sun.management.jmxremote.authenticate=true \ -
> Dcom.sun.management.jmxremote.password.file=/usr/local/tomcat/conf/jmxr
> emote.password \ -
> Dcom.sun.management.jmxremote.access.file=/usr/local/tomcat/conf/jmxrem
> ote.access \ -Djavax.net.debug=SSL \ -
> Djavax.net.ssl.trustStore=/usr/local/tomcat/certs/servertrust.jks \ -
> Djavax.net.ssl.trustStorePassword=CrazyPasswordHere \ -
> Djavax.net.ssl.keyStore=/usr/local/tomcat/certs/serverclient.jks \ -
> Djavax.net.ssl.keyStorePassword=CrazyPasswordHere '
> Environment='

RE: Apache Tomcat Version 7.0.59 on CentOS 6.8 to handle 2k requests/second

[John.E.Gregg]

> -Original Message-
> From: Kaushal Shriyan [mailto:kaushalshri...@gmail.com]
> Sent: Monday, November 14, 2016 8:46 AM
> To: Tomcat Users List
> Subject: Re: Apache Tomcat Version 7.0.59 on CentOS 6.8 to handle 2k
> requests/second
> 
> On Mon, Nov 14, 2016 at 8:06 PM, Mark Thomas  wrote:
> 
> > On 14/11/2016 14:28, Kaushal Shriyan wrote:
> > > Hi,
> > >
> > > is there a way to configure tomcat to handle 2k requests/second (2k
> > meaning
> > > 2000 requests per second). How many cpu cores do i need to setup
> > > tomcat
> > to
> > > handle 2k requests/second? How much physical memory the server
> > > should
> > have?
> >
> > That will depend on your application.
> >
> > My laptop will easily handle 10 times that for a simple application. I
> > have seen much larger servers unable to handle more than a handful of
> > requests a second for a large, complex, poorly performing application.
> >
> > > Any guidelines or rule of thumb to be followed? Any help will be
> > > highly appreciable.
> >
> > Profile your application.
> >
> 
> Thanks Mark Thomas for the quick reply.
> 
> Please suggest me any open source profiling application which i can start 
> with. I
> will appreciate if you can help me understand the relation between 2k
> requests/second vs cpu cores of the server.
> 
> Thanks again in advance.
> 
> Regards,
> 
> Kaushal

There is no fixed relationship between requests per second and number of cores. 
 It all depends on your application.  Think of it this way: if each request 
takes 1 second, you'll get fewer requests per second than if each request takes 
1 millisecond.  Whether a request takes 1 second or 1 millisecond has more to 
do with your application than hardware.

As for an open source profiler, try asking your favorite search engine.

John

RE: Why is Tomcat sending "Connection: close?"

[John.E.Gregg]

-Original Message-
From: André Warnier (tomcat) [mailto:a...@ice-sa.com] 
Sent: Wednesday, August 31, 2016 5:25 PM
To: users@tomcat.apache.org
Subject: Re: Why is Tomcat sending "Connection: close?"

Hi.
Please do not top-post your responses on this list.
See : http://tomcat.apache.org/lists.html#tomcat-users  # 6)
>
>
> -Original Message-
> From: André Warnier (tomcat) [mailto:a...@ice-sa.com]
> Sent: Wednesday, August 31, 2016 10:53 AM
> To: users@tomcat.apache.org
> Subject: Re: Why is Tomcat sending "Connection: close?"
>
> On 31.08.2016 17:50, john.e.gr...@wellsfargo.com wrote:
>> All,
>>
>> I'm using Tomcat 7.0.70 and am having trouble understanding why Tomcat is 
>> sending "Connection: close" in the response header as often as it is.  With 
>> almost no load on the server, I get "Connection: close" pretty much every 
>> time.  The client is sending "Connection: keep-alive" but it doesn't seem to 
>> matter.  HTTP protocol is 1.1 and response code is 200.
>>
>> In other cases I've seen Tomcat behave exactly the way the doc says the 
>> below config should behave (100 requests per connection as long as the 
>> timeout is not exceeded) but not this time.
>>
>> Any idea why this is occurring or where to look to debug it?  I've tried 
>> setting breakpoints in AbstractHttp11Processor where "Connection: close" is 
>> set, but it's not hit.
>>
>> Thanks
>>
>> John
>>
>> Here is my connector config:
>>
>> >   protocol="HTTP/1.1"
>>   SSLEnabled="true"
>>   maxThreads="80"
>>   maxKeepAliveRequests="100"
>>   keepAliveTimeout="1"
>>   scheme="https"
>>   secure="true"
>>   clientAuth="true"
>>   sslProtocol="TLS"
>>   sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2,SSLv2Hello,SSLv3"
>>   keystoreFile="${keystoreFile}"
>>   keystorePass="${keystorePassword}"
>>   keyAlias="test"
>>   truststoreFile="${truststoreFile}"
>>   truststorePass="${truststorePassword}"
>>   allowUnsafeLegacyRenegotiation="false"
>>   ciphers="SSL_DH_DSS_WITH_3DES_EDE_CBC_SHA,
>> SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_ 
>> CBC_SHA, SSL_DH_RSA_WITH_3DES_EDE_CBC_SHA, 
>> SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_RC4_128_SHA, 
>> TLS_DHE_DSS_WITH_AES _128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, 
>> TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, 
>> TLS_ ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, 
>> TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,
>> TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_EC DSA_WITH_3DES_EDE_CBC_SHA, 
>> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, 
>> TLS_ECDHE_RSA_WIT H_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, 
>> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_C BC_SHA, 
>> TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, 
>> TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH _AES_256_CBC_SHA"
>>   />
>>
>
> Sorry to ask, but are you positive that there is *nothing* between the 
> browser and Tomcat ? (I mean like a firewall, proxy server, etc..)
>
> > No, unfortunately not.  I put "%{Connection}i %{Connection}o" in the 
> > log config and get
"Keep-Alive close."  IOW the client sends Keep-Alive but Tomcat responds with 
close.
>

There are a number of scenarios in which a webserver - not only Tomcat - would 
send a
"Connection: close" header in a response, and close the connection after 
sending the response.  For the gory details, see 
https://tools.ietf.org/html/rfc7230#section-6.3 etc..
Simplified :

- if a webserver knows in advance the size of the response body, then it would 
normally send a "Content-length: xxx" header, followed by a response body of 
exactly that length. 
In such a case also, it should normally honor the keep-alive request of the 
client, and not close the connection

- if the server does not know in advance the size of the response body (e.g. it 
is generated dynamically by a script or servlet or such), then it cannot send a 
Content-length header in advance, and it has 2 choices :
   a) use a "Transfer-encoding: chunked" method, whereby the response body is 
"packaged" 
in successive "chunks", each with a header giving its chunk length, the 
sequence being terminated by a zero-length chunk (which tells the client that 
this response is finished)
   or
   b) if it cannot use (or is forbidden to use) the above for some reason, then 
the only way is to send the response body as it arrives from whatever generates 
it, and when that body ends, and it has sent the whole body to the client, 
close the connection.
The closing of the connection then acts for the client as a signal that the 
response to this request is finished (as there would be no way for the client 
otherwise to know if there is more to come or not).
In such a case, if the webserver were to know in advance that this is the case 
(before sending the first chun

RE: Why is Tomcat sending "Connection: close?"

[John.E.Gregg]

No, unfortunately not.  I put "%{Connection}i %{Connection}o" in the log config 
and get "Keep-Alive close."  IOW the client sends Keep-Alive but Tomcat 
responds with close.

Thanks




-Original Message-
From: André Warnier (tomcat) [mailto:a...@ice-sa.com] 
Sent: Wednesday, August 31, 2016 10:53 AM
To: users@tomcat.apache.org
Subject: Re: Why is Tomcat sending "Connection: close?"

On 31.08.2016 17:50, john.e.gr...@wellsfargo.com wrote:
> All,
>
> I'm using Tomcat 7.0.70 and am having trouble understanding why Tomcat is 
> sending "Connection: close" in the response header as often as it is.  With 
> almost no load on the server, I get "Connection: close" pretty much every 
> time.  The client is sending "Connection: keep-alive" but it doesn't seem to 
> matter.  HTTP protocol is 1.1 and response code is 200.
>
> In other cases I've seen Tomcat behave exactly the way the doc says the below 
> config should behave (100 requests per connection as long as the timeout is 
> not exceeded) but not this time.
>
> Any idea why this is occurring or where to look to debug it?  I've tried 
> setting breakpoints in AbstractHttp11Processor where "Connection: close" is 
> set, but it's not hit.
>
> Thanks
>
> John
>
> Here is my connector config:
>
>   protocol="HTTP/1.1"
>  SSLEnabled="true"
>  maxThreads="80"
>  maxKeepAliveRequests="100"
>  keepAliveTimeout="1"
>  scheme="https"
>  secure="true"
>  clientAuth="true"
>  sslProtocol="TLS"
>  sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2,SSLv2Hello,SSLv3"
>  keystoreFile="${keystoreFile}"
>  keystorePass="${keystorePassword}"
>  keyAlias="test"
>  truststoreFile="${truststoreFile}"
>  truststorePass="${truststorePassword}"
>  allowUnsafeLegacyRenegotiation="false"
>  ciphers="SSL_DH_DSS_WITH_3DES_EDE_CBC_SHA, 
> SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_ CBC_SHA, 
> SSL_DH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, 
> SSL_RSA_WITH_RC4_128_SHA, TLS_DHE_DSS_WITH_AES _128_CBC_SHA, 
> TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, 
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ 
> ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, 
> TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_EC DSA_WITH_3DES_EDE_CBC_SHA, 
> TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, 
> TLS_ECDHE_RSA_WIT H_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, 
> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_C BC_SHA, 
> TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, 
> TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH _AES_256_CBC_SHA"
>  />
>

Sorry to ask, but are you positive that there is *nothing* between the browser 
and Tomcat ? (I mean like a firewall, proxy server, etc..)



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Why is Tomcat sending "Connection: close?"

[John.E.Gregg]

All,

I'm using Tomcat 7.0.70 and am having trouble understanding why Tomcat is 
sending "Connection: close" in the response header as often as it is.  With 
almost no load on the server, I get "Connection: close" pretty much every time. 
 The client is sending "Connection: keep-alive" but it doesn't seem to matter.  
HTTP protocol is 1.1 and response code is 200.

In other cases I've seen Tomcat behave exactly the way the doc says the below 
config should behave (100 requests per connection as long as the timeout is not 
exceeded) but not this time.

Any idea why this is occurring or where to look to debug it?  I've tried 
setting breakpoints in AbstractHttp11Processor where "Connection: close" is 
set, but it's not hit.

Thanks

John

Here is my connector config: