Re: Tomcat 8 APR/openSSL Issue

[Syam Pillai]

Dear Mark,
Thanks for the response.

I already have the following installed:

Tomcat Native: 1.2.10-1.18.amzn1
OpenSSL 1.0.2k-fips  26 Jan 2017

Also, please see this:
ld /usr/lib64/libtcnative-1.so.0.2.10
ld: warning: cannot find entry symbol _start; not setting start address
/usr/lib64/libtcnative-1.so.0.2.10: undefined reference to
`SSL_CTX_add0_chain_cert'

The Tomcat Native package is from Amazon (I'm on their AMI Linux server).
Is this a packaging problem from their side?


On Fri, Oct 6, 2017 at 6:02 PM, Mark Thomas  wrote:

> On 05/10/17 22:10, Syam Pillai wrote:
> > On my AMI (Amazon Linux) server, tomcat 8 was running happily but today,
> > after an upgrade (Version is now 8.5.16.0), the server is failing to
> start
> > with the following message:
> >
> > INFO [main] org.apache.coyote.AbstractProtocol.init Initializing
> > ProtocolHandler ["https-openssl-nio-8443"]
> > /usr/share/soengine/jdk/bin/java: symbol lookup error:
> > /usr/lib64/libtcnative-1.so.0.2.10: undefined symbol:
> > SSL_CTX_add0_chain_cert
> >
> > I can see that before these lines,
> > OpenSSL is loaded:
> > INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL
> > OpenSSL successfully initialized [OpenSSL 1.0.1e-fips 11 Feb 2013]
>
> Tomcat 8.5.x requires Tomcat Native 1.2.x
>
> Tomcat Native 1.2.x requires OpenSSL 1.0.2
>
> The above is never going to work.
>
> > However,
> > ​ ​
> > I don't know why this version of OpenSSL is being shown. On the OS
> terminal
> > (Kernel:
> > 4.9.51-10.52.amzn1.x86_64 #1 SMP), if I check, it is showing a different
> > version. (I could not find any duplicate installation of OpenSSL
> > ​ ​
> > on the server).
> >
> > openssl version -v
> > OpenSSL 1.0.2k-fips  26 Jan 2017
>
> You need to contact the provider of your Tomcat Native binary.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>


-- 
*Syam S. Pillai, **Director & Chief Technology Officer*
*ENGRAV Aviation Services & Systems Pvt. Ltd.*
*# 15, Level 1, Indradhanush,  Gubbi Cross,*
*Kothannur PO, Bangalore - 560 077, India.*
*Phone: +91 80 2844 3740*
*http://www.engravgroup.com *

Re: BREAKTHROUGH (but not solved) Re: Problem: (GSKit) No compatible cipher suite available between SSL end points.

[logo]

James,


> On 10/6/17, 6:58 AM, Mark Thomas (Tomcat List) wrote:
> 
>> It might help to think of it like this:
>> 
>> There are the ciphers that a JVM supports.
>> The JVM only enables sub-set of the supported ciphers are enabled by
>> default.
>> Tomcat with a default configuration only uses a sub-set of the ciphers
>> that the JVM enables by default.
>> . . .
>> It looks like you have an incompatible set of ciphers configured.
>> 
>> As per Chris's previous email, it looks like RSA_AES_256_CBC_SHA is the
>> least worse option. The Java name for this is:
>> TLS_RSA_WITH_AES_256_CBC_SHA
> 
> I should have tried this DAYS ago. There is also a Tomcat 7 server installed 
> on the Google Cloud server. With no apparent differences in the Java list of 
> available and "enabled-by-default" ciphers between the two boxes, it's clear 
> that the biggest single difference that I'm actually able to do anything 
> about is which Tomcat server is running on 443.
> 
> So with both Tomcat servers shut down, I switched Tomcat 7 over to port 443, 
> brought it up, and tried connecting to it from the same program as before.
> 
> This time, I got a 404. Not the least bit surprising, since the webapp 
> context isn't actually installed on the Tomcat 7 server.
> 
> Incidentally, I also tried running the ssllabs.com  test 
> on the Tomcat 7 server. The results weren't very meaningful: it only listed 
> the ECDHE suites, but then again, it only listed the ECDHE suites when I 
> tried it on one of our other Tomcat 7 servers.
> 
> > Tomcat with a default configuration only uses a sub-set of the ciphers
> > that the JVM enables by default.
> 
> So is there a way, short of downloading and recompiling Tomcat myself, to 
> control what's in that default subset of a default subset?
> 
> Or failing that, is there a way, in my connector tag, to specify "Use 
> TLS_RSA_WITH_AES_256_CBC_SHA in addition to all the suites Tomcat 8.5 uses by 
> default"?
> 
> Or do I need to list all the Tomcat 8.5 defaults in a "ciphers" clause, along 
> with the TLS_RSA_WITH_AES_256_CBC_SHA?
> 
> Noting that my connector tag is written using Tomcat 7 connector syntax, is 
> there a good example of how to code a ciphers clause for that tag?
> 
> --
> JHHL
> 

Apparently Tomcat 7 syntax will not support openssl syntax like 
HIGH:+AES256:!MD5:... That will perfectly work with Tomcat 8.5 with the new ssl 
syntax, and that is a major achievement.

A quick dig into tomcat7 conf and tested with docker will lead you to the 
following working ssl conf. Be aware that Java7 is not really able to handle 
modern ssl requirements like bit length > 1024bit or server cipher order.



You may want to test a local ssl setup with the great testssl.sh tool by dirk 
wetter (https://testssl.sh ), also available on docker.
I’ve picked the ciphers from the list that testssl.sh showed me for the 
standard conf of Tomcat 8.0.47 and JRE7 (RFC notation). Then I’ve added your 
requested cipher.

HTH,

Peter

BREAKTHROUGH (but not solved) Re: Problem: (GSKit) No compatible cipher suite available between SSL end points.

[James H. H. Lampert]

On 10/6/17, 6:58 AM, Mark Thomas (Tomcat List) wrote:


It might help to think of it like this:

There are the ciphers that a JVM supports.
The JVM only enables sub-set of the supported ciphers are enabled by
default.
Tomcat with a default configuration only uses a sub-set of the ciphers
that the JVM enables by default.
. . .
It looks like you have an incompatible set of ciphers configured.

As per Chris's previous email, it looks like RSA_AES_256_CBC_SHA is the
least worse option. The Java name for this is:
TLS_RSA_WITH_AES_256_CBC_SHA


I should have tried this DAYS ago. There is also a Tomcat 7 server 
installed on the Google Cloud server. With no apparent differences in 
the Java list of available and "enabled-by-default" ciphers between the 
two boxes, it's clear that the biggest single difference that I'm 
actually able to do anything about is which Tomcat server is running on 443.


So with both Tomcat servers shut down, I switched Tomcat 7 over to port 
443, brought it up, and tried connecting to it from the same program as 
before.


This time, I got a 404. Not the least bit surprising, since the webapp 
context isn't actually installed on the Tomcat 7 server.


Incidentally, I also tried running the ssllabs.com test on the Tomcat 7 
server. The results weren't very meaningful: it only listed the ECDHE 
suites, but then again, it only listed the ECDHE suites when I tried it 
on one of our other Tomcat 7 servers.


> Tomcat with a default configuration only uses a sub-set of the ciphers
> that the JVM enables by default.

So is there a way, short of downloading and recompiling Tomcat myself, 
to control what's in that default subset of a default subset?


Or failing that, is there a way, in my connector tag, to specify "Use 
TLS_RSA_WITH_AES_256_CBC_SHA in addition to all the suites Tomcat 8.5 
uses by default"?


Or do I need to list all the Tomcat 8.5 defaults in a "ciphers" clause, 
along with the TLS_RSA_WITH_AES_256_CBC_SHA?


Noting that my connector tag is written using Tomcat 7 connector syntax, 
is there a good example of how to code a ciphers clause for that tag?


--
JHHL

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance settings for Multiple Hosts

[Jerry Malcolm]

On 10/6/2017 5:15 AM, André Warnier (tomcat) wrote:

On 06.10.2017 11:57, Suvendu Sekhar Mondal wrote:
On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm 
 wrote:
I am running TC 8.0 on WinServer8 on a commercially hosted platform 
with a

WAMP environment.   I am running around 10 virtual hosts.   2 hosts are
dedicated to JSPWiki.  The other 8 are running variations of the 
same custom

application with around 10-15 individual webapps each.

When I am running 7 of the 10 hosts, performance is great.  I get JSP
response time under a second.  But when I add just a couple of more 
of the

hosts, my page response time on all of the apps goes from an acceptable
under a second to horrible at around 15-20 seconds per page. It doesn't
seem to be a specific host causing the problem.  Reducing overall 
hosts in

any order makes the problem go away.

I've looked at the the processor utilization during the good times 
and slow
times, and don't see a significant difference.  I have 16GB of 
memory, and
it consistently shows about 35% utilization.  I also checked mySQL 
response
time, and the per-query SQL response does not vary.  So it doesn't 
appear to

be a db problem.

I suspect there is some TC configuration parameter such as heap, etc 
that I
need to tweak. (But I'm not getting OutOfMemory errors).  But I 
don't know
which one, and I don't know a formula to use to figure out what to 
set it
to.  So I just need a little education.  What tools can I use to 
help me
figure out what is going south slowing everything to a crawl when 
the extra

hosts come online?  And what parameters should I be looking at (and how
should I calculate the proper values based on number of hosts)?

Suggestions?


Interesting problem. Couple of questions:
Is the slowness sporadic or persistent? What happens when you again
shutdown some of the hosts? Does response time comes back to normal?
Is all transactions are slow or some of them which are routed to
specific host(s)?
Are you fronting Tomcat instances with Apache? If yes, then please
post BalancerMember configuration of the Tomcat cluster here.



Taking the OP's observations above at face value, I think that first 
thing that I would do, is to configure the Java JVM that runs this 
tomcat, to do some logging of the Garbage Collection events. And then 
I would look at that logfile, and find out if there is some 
significant difference (in frequency of GC, or in the mounts of memory 
mentioned for each event), depending on the number of Hosts enabled.


You would have to look at the appropriate command-line switches to use 
for the particular JVM being used, for how to set this up.
And then (since this seems to be on a Windows platform, running tomcat 
as a service), you should use the "tomcat(version)w.exe" program (a 
GUI), to set these parameters properly.

See e.g.
https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html#Tomcat_service_application 



--> "Tomcat monitor application"


Thanks for the responses. Answers to several questions in responses:

1) The 16GB is physical OS memory, not JVM memory
2) Re: 'balancermember config' I am running one instance of Apache 
with mod_jk to one one instance of Tomcat containing all of the 10 
virtual hosts.  I'm not familiar with 'balancermember'.  But it sounds 
like something to do with a cluster config.  If it still applies to my 
config, let me know where I can learn more about it.

3) I'll look into turning on garbage collection logging
4) The problem is constant when I have 'too many' hosts.  And the 
problem is pervasive to all running hosts. I've found the magic number 
is 7 hosts.  7 hosts and life is good.  8+ hosts  and everything clogs 
up big-time.


I failed to mention that none of these host sites are anywhere close to 
'high-volume'.  When a user hits the site, there will be flurry of page 
activity for a few minutes.  But I'd say each of the sites averages one 
or two user hits per hour on average.  So bringing sites 8, 9, and 10 
online is not bringing massive additional tomcat/mySQL/network 
activity.  It's basically bringing 3 more 'effectively dormant' web 
sites online.  This seems to me that it's got to have something to with 
the size of the basic inactive footprint of each host.


Jerry


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: ISAPI and IIS 10 Logging Issue

[Mark Thomas]

On 06/10/17 16:27, Mark Thomas wrote:
>> On 10.05.2017 8:54, Thomas, Michael wrote:
>>
>> Unfortunately I am not getting much traction with Microsoft.  From the IIS 
>> forum, it looks like they are pointing the finger in the direction of the 
>> "third-party" that is writing the module.
> 
> That is consistent with my experience with MS support.
> 
>> This may be an obvious question but has the team who created the ISAPI 
>> module tested it on Windows Server 2016?
> 
> Probably not. I took on the release manager role for 1.2.42 and from
> memory I was testing with 2008.
> 
>>  If so, what do your IIS logs look like?  I can submit anonymized IIS logs 
>> if that helps.  I also analyzed IIS trace logs for a request using the ISAPI 
>> redirect on both Windows 2016 and Windows 2012 R2.  There are no discernable 
>> differences.
>>
>> Can anyone confirm the functionality in Windows Server 2016?
> 
> I'm setting up a 2016 VM now. I'll report back when I get some results.

I see the same as you.

> I did find this:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=45769
> 
> which sounds suspiciously similar to your report.
> 
> I've tried and failed to find the referenced KB article.
> 
> The docs for setting up IIS are rather out of date. Providing some
> updated instructions looks like an opportunity to contribute if you (or
> anyone else) is interested.

I'll look at the IIS API docs next week and see if there is something
different we should be doing in IIS 10 compared to 8.5 and earlier.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: ISAPI and IIS 10 Logging Issue

[Igal @ Lucee.org]

On 10/6/2017 8:27 AM, Mark Thomas wrote:

On 10.05.2017 8:54, Thomas, Michael wrote:

Unfortunately I am not getting much traction with Microsoft.  From the IIS forum, it 
looks like they are pointing the finger in the direction of the "third-party" 
that is writing the module.

That is consistent with my experience with MS support.


I had the displeasure of working with IIS for many years until one day I 
switched to nginx and never looked back.  Works very well with the http 
connector and the RemoteIpFilter.


TBH, I am currently looking at HAProxy as an alternative to nginx, but 
that's for other reasons.


My 2c,

Igal Sapir
Lucee Core Developer
Lucee.org 

Re: ISAPI and IIS 10 Logging Issue

[Mark Thomas]

> On 10.05.2017 8:54, Thomas, Michael wrote:
> 
> Unfortunately I am not getting much traction with Microsoft.  From the IIS 
> forum, it looks like they are pointing the finger in the direction of the 
> "third-party" that is writing the module.

That is consistent with my experience with MS support.

> This may be an obvious question but has the team who created the ISAPI module 
> tested it on Windows Server 2016?

Probably not. I took on the release manager role for 1.2.42 and from
memory I was testing with 2008.

>  If so, what do your IIS logs look like?  I can submit anonymized IIS logs if 
> that helps.  I also analyzed IIS trace logs for a request using the ISAPI 
> redirect on both Windows 2016 and Windows 2012 R2.  There are no discernable 
> differences.
> 
> Can anyone confirm the functionality in Windows Server 2016?

I'm setting up a 2016 VM now. I'll report back when I get some results.

I did find this:
https://bz.apache.org/bugzilla/show_bug.cgi?id=45769

which sounds suspiciously similar to your report.

I've tried and failed to find the referenced KB article.

The docs for setting up IIS are rather out of date. Providing some
updated instructions looks like an opportunity to contribute if you (or
anyone else) is interested.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8 APR/openSSL Issue

[Mark Thomas]

On 05/10/17 22:10, Syam Pillai wrote:
> On my AMI (Amazon Linux) server, tomcat 8 was running happily but today,
> after an upgrade (Version is now 8.5.16.0), the server is failing to start
> with the following message:
> 
> INFO [main] org.apache.coyote.AbstractProtocol.init Initializing
> ProtocolHandler ["https-openssl-nio-8443"]
> /usr/share/soengine/jdk/bin/java: symbol lookup error:
> /usr/lib64/libtcnative-1.so.0.2.10: undefined symbol:
> SSL_CTX_add0_chain_cert
> 
> I can see that before these lines,
> OpenSSL is loaded:
> INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL
> OpenSSL successfully initialized [OpenSSL 1.0.1e-fips 11 Feb 2013]

Tomcat 8.5.x requires Tomcat Native 1.2.x

Tomcat Native 1.2.x requires OpenSSL 1.0.2

The above is never going to work.

> However,
> ​ ​
> I don't know why this version of OpenSSL is being shown. On the OS terminal
> (Kernel:
> 4.9.51-10.52.amzn1.x86_64 #1 SMP), if I check, it is showing a different
> version. (I could not find any duplicate installation of OpenSSL
> ​ ​
> on the server).
> 
> openssl version -v
> OpenSSL 1.0.2k-fips  26 Jan 2017

You need to contact the provider of your Tomcat Native binary.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Problem: (GSKit) No compatible cipher suite available between SSL end points.

[Mark Thomas]

On 05/10/17 18:52, James H. H. Lampert wrote:
> This just keeps getting weirder.
> 
> Late yesterday afternoon, I did a lengthy "stare-and-compare" between
> what SSLInfo returned for the two different Tomcat servers, and I
> couldn't find any differences. But then, I got called away from this on
> something that kept me in the office until after 7 PM.
> 
> Finally getting back to it, I looked at the "connector ciphers" on the
> Tomcat 8 manager (there isn't one on the Tomcat 7 manager), and saw that
> only 16 of the 36 ciphers that SSLInfo starred as "default" are actually
> enabled in Tomcat.

It might help to think of it like this:

There are the ciphers that a JVM supports.
The JVM only enables sub-set of the supported ciphers are enabled by
default.
Tomcat with a default configuration only uses a sub-set of the ciphers
that the JVM enables by default.

This is a little dated but might help:
https://wiki.apache.org/tomcat/Security/Ciphers

> Then, using what Mr. Schultz told me about reading cipher names, I
> compared what actually *does* come up in the Tomcat 8 manager with the
> DSPSYSVAL on the AS/400. And I found that if
>> *RSA_AES_256_CBC_SHA
> is the same as
>> TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
> 
> then maybe we DO have a common cipher, at least in theory (unless
> "ECDHE" makes it otherwise).

ECDHE makes them different.

> Unfortunately, I can't run the local box's Tomcat server through
> SSLLabs, because it's on a nonstandard port number, and Tomcat 7 doesn't
> have a "connector ciphers" button on the manager main page.
> 
> The cloud box is a Google Compute Engine instance. Is it possible that
> Google is somehow vetoing the handshake?

It looks like you have an incompatible set of ciphers configured.

As per Chris's previous email, it looks like RSA_AES_256_CBC_SHA is the
least worse option. The Java name for this is:
TLS_RSA_WITH_AES_256_CBC_SHA

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Severity of CVE-2016-6816

[Justin]

 Great, thank you for answers Mark. Much appreciated!

On Thursday, October 5, 2017, 3:29:04 PM CDT, Mark Thomas 
 wrote:  
 
 On 05/10/17 20:35, Justin wrote:
> Hi all,
> I'm curious about the severity of CVE-2016-6816 fixed last November.
> http://svn.apache.org/viewvc?view=revision=1767653
> 
> I see that vendors backported this fix (over time).
> http://www.securitytracker.com/id/1037332
> Severity is described as Important, Moderate, CVSS v2 Medium, and CVSS v3 
> High.
> "A remote user can exploit this to spoof content on the target server, 
> attempt to poison any intermediate web caches, or conduct cross-site 
> scripting attacks."
> This sounds pretty serious. If my server is running a vulnerable version of 
> Tomcat, how easy is it to exploit?

I'm not going to go into the details of exactly how to exploit this
issue but the answer to your question is pretty difficult.

1. You need to be running Tomcat behind a reverse proxy of some form.

2. That reverse proxy needs to accept the same invalid requests as
  Tomcat (it should reject them with a 400 response).

3. The attacker needs to be able to either upload arbitrary static
  content which they can then request or formulate a request in such a
  way that arbitrary content is injected into the response.

1 is very common. 2 is possible but I haven't seen any evidence of it
for these particular invalid requests. 3 is (very) unlikely.

Overall, I'd judge the chances of all of these lining up on any given
system as low to very low. It is one of those where the chances of it
happening are low but the impact if it does happen is high. You could
argue rating of Low, Moderate or High for this on Tomcat's scale. It is
very much a judgement call. Where there is uncertainty over the
likelihood then we tend to err on the side of caution.

> Must webapps use cookies as some example exploits have shown?

No.

> "By manipulating the HTTP response the attacker could... obtain sensitive 
> information from requests other then their own." So if trusted users are 
> logging in, untrusted users could obtain passwords? Path or query parameters?

Untrusted users could obtain anything from the response intended for a
trusted user. Passwords are therefore unlikely, as is anything on the
request side unless the application is echoing that data back to the
client. Session cookies are probably the most likely target.

HTH,

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

  

Re: Tomcat 8.5.23 - Nio2 Connector - Error reading request, ignored

[Rémy Maucherat]

On Thu, Oct 5, 2017 at 7:11 PM, Eirik Lykken 
wrote:

>
> 05-Oct-2017 17:13:32.535 SEVERE 
> [https-openssl-nio2-94.229.64.230-443-exec-19701]
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process Error
> reading request, ignored
>  java.lang.IllegalStateException
> at org.apache.coyote.http2.Http2UpgradeHandler.fill(Http2Upgrad
> eHandler.java:1314)
> at org.apache.coyote.http2.Http2UpgradeHandler.fill(Http2Upgrad
> eHandler.java:1290)
> at org.apache.coyote.http2.Http2Parser.readConnectionPreface(Ht
> tp2Parser.java:574)
> at org.apache.coyote.http2.Http2UpgradeHandler.init(Http2Upgrad
> eHandler.java:243)
> at org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(
> Http2UpgradeHandler.java:310)
> at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.di
> spatch(UpgradeProcessorInternal.java:54)
> at org.apache.coyote.AbstractProcessorLight.process(AbstractPro
> cessorLight.java:53)
> at org.apache.coyote.AbstractProtocol$ConnectionHandler.process
> (AbstractProtocol.java:868)
> at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRu
> n(Nio2Endpoint.java:1693)
> at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketPro
> cessorBase.java:49)
> at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(Ab
> stractEndpoint.java:946)
> at org.apache.tomcat.util.net.SecureNio2Channel$HandshakeWriteC
> ompletionHandler.completed(SecureNio2Channel.java:115)
> at org.apache.tomcat.util.net.SecureNio2Channel$HandshakeWriteC
> ompletionHandler.completed(SecureNio2Channel.java:108)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
> at sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixA
> synchronousSocketChannelImpl.java:736)
> at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousS
> ocketChannelImpl.java:382)
> at sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousS
> ocketChannelImpl.java:399)
> at java.nio.channels.AsynchronousSocketChannel.write(Asynchrono
> usSocketChannel.java:577)
> at org.apache.tomcat.util.net.SecureNio2Channel.handshakeIntern
> al(SecureNio2Channel.java:273)
> at org.apache.tomcat.util.net.SecureNio2Channel.handshake(Secur
> eNio2Channel.java:204)
> at org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRu
> n(Nio2Endpoint.java:1671)
> at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketPro
> cessorBase.java:49)
> at org.apache.tomcat.util.net.AbstractEndpoint.processSocket(Ab
> stractEndpoint.java:946)
> at org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCo
> mpletionHandler.completed(SecureNio2Channel.java:98)
> at org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCo
> mpletionHandler.completed(SecureNio2Channel.java:91)
> at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> at sun.nio.ch.Invoker$2.run(Invoker.java:218)
> at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousCh
> annelGroupImpl.java:112)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:624)
> at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.r
> un(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:748)
>

I think the state of the handshake is most likely ok despite the weird
looking stack. The code doing blocking SSL reads is complex also, it would
be returning 0 bytes for a blocking read in this case. To examine the issue
it would need to be reproduced. Although it might simply be an IO error
that doesn't get reported as it should and you see that stack instead (but
the result is then the same).


>
> This is our current Connector configuration:
>
>  protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> SSLEnabled="true"
> maxThreads="1500" acceptCount="5000" scheme="https" secure="true"
> defaultSSLHostConfigName="xxx"
> hostName="xxx"
> connectionTimeout="7"
> URIEncoding="UTF-8"
> server="Apache"
> enableLookups="false"
> compression="on" compressionMinSize="2048"
> compressableMimeType="text/html,text/xml,text/csv,text/css,t
> ext/javascript,text/html,text/plain,application/javascript,a
> pplication/x-javascript"
> sslImplementationName="org.apache.tomcat.util.net.openssl.Op
> enSSLImplementation"
>
> > readTimeout="7" writeTimeout="7" maxConcurrentStreams="400"
> maxConcurrentStreamExecution="400"/>
>

maxConcurrentStreamExecution="400" that's way too much, don't do that.


> We have experimented with different parameters of the Http2Protocol
> connector to improve performance and control the leak with
> Http11NioProtocol connector but the exceptions are 

Re: Performance settings for Multiple Hosts

[tomcat]

On 06.10.2017 11:57, Suvendu Sekhar Mondal wrote:

On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm  wrote:

I am running TC 8.0 on WinServer8 on a commercially hosted platform with a
WAMP environment.   I am running around 10 virtual hosts.   2 hosts are
dedicated to JSPWiki.  The other 8 are running variations of the same custom
application with around 10-15 individual webapps each.

When I am running 7 of the 10 hosts, performance is great.  I get JSP
response time under a second.  But when I add just a couple of more of the
hosts, my page response time on all of the apps goes from an acceptable
under a second to horrible at around 15-20 seconds per page.  It doesn't
seem to be a specific host causing the problem.  Reducing overall hosts in
any order makes the problem go away.

I've looked at the the processor utilization during the good times and slow
times, and don't see a significant difference.  I have 16GB of memory, and
it consistently shows about 35% utilization.  I also checked mySQL response
time, and the per-query SQL response does not vary.  So it doesn't appear to
be a db problem.

I suspect there is some TC configuration parameter such as heap, etc that I
need to tweak. (But I'm not getting OutOfMemory errors).  But I don't know
which one, and I don't know a formula to use to figure out what to set it
to.  So I just need a little education.  What tools can I use to help me
figure out what is going south slowing everything to a crawl when the extra
hosts come online?  And what parameters should I be looking at (and how
should I calculate the proper values based on number of hosts)?

Suggestions?


Interesting problem. Couple of questions:
Is the slowness sporadic or persistent? What happens when you again
shutdown some of the hosts? Does response time comes back to normal?
Is all transactions are slow or some of them which are routed to
specific host(s)?
Are you fronting Tomcat instances with Apache? If yes, then please
post BalancerMember configuration of the Tomcat cluster here.



Taking the OP's observations above at face value, I think that first thing that I would 
do, is to configure the Java JVM that runs this tomcat, to do some logging of the Garbage 
Collection events. And then I would look at that logfile, and find out if there is some 
significant difference (in frequency of GC, or in the mounts of memory mentioned for each 
event), depending on the number of Hosts enabled.


You would have to look at the appropriate command-line switches to use for the particular 
JVM being used, for how to set this up.
And then (since this seems to be on a Windows platform, running tomcat as a service), you 
should use the "tomcat(version)w.exe" program (a GUI), to set these parameters properly.

See e.g.
https://tomcat.apache.org/tomcat-8.5-doc/windows-service-howto.html#Tomcat_service_application

--> "Tomcat monitor application"





-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Performance settings for Multiple Hosts

[Suvendu Sekhar Mondal]

On Fri, Oct 6, 2017 at 2:06 AM, Jerry Malcolm  wrote:
> I am running TC 8.0 on WinServer8 on a commercially hosted platform with a
> WAMP environment.   I am running around 10 virtual hosts.   2 hosts are
> dedicated to JSPWiki.  The other 8 are running variations of the same custom
> application with around 10-15 individual webapps each.
>
> When I am running 7 of the 10 hosts, performance is great.  I get JSP
> response time under a second.  But when I add just a couple of more of the
> hosts, my page response time on all of the apps goes from an acceptable
> under a second to horrible at around 15-20 seconds per page.  It doesn't
> seem to be a specific host causing the problem.  Reducing overall hosts in
> any order makes the problem go away.
>
> I've looked at the the processor utilization during the good times and slow
> times, and don't see a significant difference.  I have 16GB of memory, and
> it consistently shows about 35% utilization.  I also checked mySQL response
> time, and the per-query SQL response does not vary.  So it doesn't appear to
> be a db problem.
>
> I suspect there is some TC configuration parameter such as heap, etc that I
> need to tweak. (But I'm not getting OutOfMemory errors).  But I don't know
> which one, and I don't know a formula to use to figure out what to set it
> to.  So I just need a little education.  What tools can I use to help me
> figure out what is going south slowing everything to a crawl when the extra
> hosts come online?  And what parameters should I be looking at (and how
> should I calculate the proper values based on number of hosts)?
>
> Suggestions?
>
Interesting problem. Couple of questions:
Is the slowness sporadic or persistent? What happens when you again
shutdown some of the hosts? Does response time comes back to normal?
Is all transactions are slow or some of them which are routed to
specific host(s)?
Are you fronting Tomcat instances with Apache? If yes, then please
post BalancerMember configuration of the Tomcat cluster here.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org