Re: Weirdest Tomcat Behavior Ever?

[Christopher Schultz]

Mark,

On 11/25/20 22:30, Christopher Schultz wrote:

Mark,

On 11/25/20 12:49, Mark Thomas wrote:

On 25/11/2020 16:46, Christopher Schultz wrote:




Strange that the library is getting a direct reference to the response's
file descriptor. That seems (to me) to be a tactical mistake in an
implementation. Then again, I'm not looking at what is actually being
accomplished, there... it may make perfect sense in context.


The library creates a pipe and so gets two file descriptors. It closes
the file descriptors so they are returned to the OS's pool of file
descriptors. Shortly afterwards the OS reuses one of the file
descriptors and assigns it to the socket. So far, so good. Then the
library closes the file descriptors for a second time and that closes
the socket.


That would do it for most situations, but this is a child process 
closing a file descriptor twice. It shouldn't break the parent process, 
should it? I mean, if I could write a program that could steal fds from 
its parent process, that would be a pretty big security bug.


Or is the issue that this is actually a "thread process" and not a 
completely separate process, so they share a fd pool?


The more I think about this the less it makes sense unless I'm missing 
something.


I want to go back to your earlier statement:

"

strace shows the .so file being accessed, then we see the process
forking, a pipe set up between the parent and child and the child
calling execve on the same .so file.

"

So this is a typical fork-and-exec to start another process. The child 
inherits the fds from the parent. Calling execve means that the child 
process is replaced so this isn't a "thread" if it somehow started that 
way. (Specifically, this means that we aren't sharing a fd *table* 
between processes, even if some fds are actually mapped to the same OS 
fd between processes.)


The non-JVM native library is being invoked, and it's decided to 
fork-off a process ... of itself? Hmm... that's odd in a JVM 
environment, but ... okay, I guess.


"

After a normal clean-up the parent then calls close on the two file
descriptors associated with the pipe for a second time."


So the child cleans them up AND the parent cleans them up? Or the parent 
cleans when up twice? The child should be able to call close() as many 
times as it wants and only poison itself. Does the child process ever 
exit()?


The parent process must be the JVM process, right? And the parent 
process (this native library, running within the JVM process) 
double-closes file descriptors, with some measurable delay? That's the 
only way this could make sense. And of course it mess mess everything up 
in really *really* unpredictable ways.


-chris

PS There are some cases where calling close(int) more than one time 
actually makes sense, but it's very rare. Evidently, HP-UX requires this 
if close() fails in certain ways, while most other systems prohibit 
double-calls to close() even when a failure occurs. So much for 
portability. Anyway, my guess is that this isn't an HP-UX library ported 
to Linux with a logic flaw introduced by that port. It's more likely 
just an implementation bug.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3

[Christopher Schultz]

Amit,

On 11/25/20 12:40, Amit Pande wrote:

Thank you Chris for the inputs. Admittedly, I didn’t know the internals of Sun 
JCE/JSSE vs BC JCE.

Pasting sample output and it indeed is taking minutes on RHEL 7.3. Not sure if 
I am indeed missing some trick here.


RHEL 7.3 ---

test-host-1:~ # date ; /usr/openv/java/jre/bin/keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass 
org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair  -keyalg RSA -keypass "Test123" 
-validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore 
"/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date

Wed Nov 25 10:52:56 CST 2020   (START TIME)

Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) 
with a validity of 3,650 days
 for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org
[Storing /tmp/test_bc.bcfks]

Wed Nov 25 10:58:11 CST 2020 (END TIME) (Almost 6 minutes)

test-host-1:~ # uname -a
Linux test-host-1 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 
x86_64 x86_64 x86_64 GNU/Linux
test-host-1:~ # cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)
test-host-1:~ #
test-host-1:~ # /usr/openv/java/jre/bin/java -version
java version "1.8.0_271"
Java(TM) SE Runtime Environment (build 1.8.0_271-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)
test-host-1:~ #


RHEL 7.2 -

[root@test-host-2 ~]# date ; /usr/openv/java/jre/bin/keytool -providerpath /root/bc-fips-1.0.2.jar -providerclass 
org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair  -keyalg RSA -keypass "Test123" 
-validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore 
"/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date

Wed Nov 25 11:20:06 CST 2020 (START TIME)

Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) 
with a validity of 3,650 days
 for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org
[Storing /tmp/test_bc.bcfks]

Wed Nov 25 11:20:10 CST 2020 (END TIME)(Almost 4 seconds)

[root@test-host-2 ~]# uname -a
Linux test-host-2  3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 
x86_64 x86_64 x86_64 GNU/Linux
[root@test-host-2 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@test-host-2 ~]#

Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@test-host-2 ~]# /usr/openv/java/jre/bin/java -version
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b33)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b33, mixed mode)
[root@test-host-2 ~]#


Since the keytool is literally taking minutes (specifically on RHEL
7.3 as you can see above), enabling FIPS OOTB has become a challenge
as it has resulted in some our test suites timing out.
What I see is that on two servers were everything is the same (except 
for the RHEL version... weird; RHEL 7.2 and 7.3 have identical kernel 
versions?), two identical command-lines take different amounts of time. 
In both cases, you are using BC with FIPS enabled.


The reason is that one of the servers is starved for entropy and the 
other one is not. Some operations require a LOT of entropy (like 
generating an RSA key) and others require less entropy, but each of 
those operations consumes some of the system's entropy as they occur.


When you run out, processes block waiting for it, and it's only created 
slowly as random hardware events are sampled to generate that "high 
quality randomness". So it can take a while.


Some servers have hardware that generates entropy faster, some servers 
have usage profiles that cause certain hardware events to occur more 
often and therefore keep the entropy pool full. Others are constantly 
starved for entropy.


Tomcat is very much a core of our product and configuration and 
starting Tomcat in timely manner (FIPS or no FIPS) has been a

critical requirement. And now, with this issue, test suites timing
out, hard to convince to get the suite timeouts increased. 😊


Well, you don't have too many choices. You either fix your entropy 
problem (which isn't exactly straightforward), or you switch to a 
process which does not require so much entropy (which may mean disabling 
FIPS), or you allow for more time to complete those operations.


Is the generation of RSA keys a part of your test suite? Maybe you can 
re-write your test suite to only generate the certificate if it's 
missing and/or is getting close to expiration. If you generate a new RSA 
key for each run of your test quite, you are certainly going to drain 
your entropy pool quickly and repeatedly.


BTW, I'm assuming that you are using (or at least just /testing/) FIPS 
mode because you need a "high security" environment. Nothing about a 
2048-bit RSA key that is valid for 10 years should be considered "high 
security".


-chris


-Original Message-
From: Christopher Schult

Re: Weirdest Tomcat Behavior Ever?

[Christopher Schultz]

Mark,

On 11/25/20 12:49, Mark Thomas wrote:

On 25/11/2020 16:46, Christopher Schultz wrote:




Strange that the library is getting a direct reference to the response's
file descriptor. That seems (to me) to be a tactical mistake in an
implementation. Then again, I'm not looking at what is actually being
accomplished, there... it may make perfect sense in context.


The library creates a pipe and so gets two file descriptors. It closes
the file descriptors so they are returned to the OS's pool of file
descriptors. Shortly afterwards the OS reuses one of the file
descriptors and assigns it to the socket. So far, so good. Then the
library closes the file descriptors for a second time and that closes
the socket.


That would do it for most situations, but this is a child process 
closing a file descriptor twice. It shouldn't break the parent process, 
should it? I mean, if I could write a program that could steal fds from 
its parent process, that would be a pretty big security bug.


Or is the issue that this is actually a "thread process" and not a 
completely separate process, so they share a fd pool?


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Weirdest Tomcat Behavior Ever?

[Mark Thomas]

On 25/11/2020 16:46, Christopher Schultz wrote:



> Strange that the library is getting a direct reference to the response's
> file descriptor. That seems (to me) to be a tactical mistake in an
> implementation. Then again, I'm not looking at what is actually being
> accomplished, there... it may make perfect sense in context.

The library creates a pipe and so gets two file descriptors. It closes
the file descriptors so they are returned to the OS's pool of file
descriptors. Shortly afterwards the OS reuses one of the file
descriptors and assigns it to the socket. So far, so good. Then the
library closes the file descriptors for a second time and that closes
the socket.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3

[Amit Pande]

Thank you Chris for the inputs. Admittedly, I didn’t know the internals of Sun 
JCE/JSSE vs BC JCE.

Pasting sample output and it indeed is taking minutes on RHEL 7.3. Not sure if 
I am indeed missing some trick here.


RHEL 7.3 ---

test-host-1:~ # date ; /usr/openv/java/jre/bin/keytool -providerpath 
/root/Downloads/bc-fips-1.0.2.jar -providerclass 
org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair  -keyalg 
RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, 
o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS 
-v -alias abcd ; date

Wed Nov 25 10:52:56 CST 2020   (START TIME)

Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) 
with a validity of 3,650 days
for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org
[Storing /tmp/test_bc.bcfks]

Wed Nov 25 10:58:11 CST 2020 (END TIME) (Almost 6 minutes)

test-host-1:~ # uname -a
Linux test-host-1 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 
x86_64 x86_64 x86_64 GNU/Linux
test-host-1:~ # cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)
test-host-1:~ #
test-host-1:~ # /usr/openv/java/jre/bin/java -version
java version "1.8.0_271"
Java(TM) SE Runtime Environment (build 1.8.0_271-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)
test-host-1:~ #


RHEL 7.2 -

[root@test-host-2 ~]# date ; /usr/openv/java/jre/bin/keytool -providerpath 
/root/bc-fips-1.0.2.jar -providerclass 
org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair  -keyalg 
RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, 
o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS 
-v -alias abcd ; date

Wed Nov 25 11:20:06 CST 2020 (START TIME)

Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) 
with a validity of 3,650 days
for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org
[Storing /tmp/test_bc.bcfks]

Wed Nov 25 11:20:10 CST 2020 (END TIME)(Almost 4 seconds)

[root@test-host-2 ~]# uname -a
Linux test-host-2  3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 
x86_64 x86_64 x86_64 GNU/Linux
[root@test-host-2 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@test-host-2 ~]#

Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@test-host-2 ~]# /usr/openv/java/jre/bin/java -version
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b33)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b33, mixed mode)
[root@test-host-2 ~]#


Since the keytool is literally taking minutes (specifically on RHEL 7.3 as you 
can see above), enabling FIPS OOTB has become a challenge as it has resulted in 
some our test suites timing out. Tomcat is very much a core of our product and 
configuration and starting Tomcat in timely manner (FIPS or no FIPS) has been a 
critical requirement. And now, with this issue, test suites timing out, hard to 
convince to get the suite timeouts increased. 😊
 

Thanks,
Amit

-Original Message-
From: Christopher Schultz  
Sent: Wednesday, November 25, 2020 10:33 AM
To: users@tomcat.apache.org
Subject: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3

Amit,

On 11/24/20 11:21, Amit Pande wrote:
> Probably not directly related to Tomcat but still sharing. Advanced 
> apologies for that.
> 
> I am using bouncy castle FIPS library and observed that specifically 
> on RHEL 7.3, the library usage is causing tremendous slowness.
Note that BC is pure-Java, so you should pretty much expect "tremendous 
slowness". The reason that the Sun JCE/JSSE provider is (or can be) very fast 
is because it's using both native code (which is only marginally faster than 
well-JIT'ed Java bytecode, if at all) and also hardware support when it's 
available. There is no way that I know of for BC's pure-Java crypto 
implementations to use hardware acceleration. AES on silicon is *much* faster 
than AES in software.

> e.g. below key tool command taking several minutes to finish.

Uhh... that's not software versus hardware. You might expect an order of 
magnitude difference between BC and hardware-accelerated SunJCE. But minutes to 
generate a 2048-bit RSA key? Something is wrong.

> keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass 
> org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair  
> -keyalg RSA -keypass "Test123" -validity 3650 -dname 
> "CN=Test_HostName, ou=My Org Unit, o=My Org" -storepass "Test123" 
> -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias test_entry
> 
> However, when I add the JVM option
> -J-Djava.security.egd=file:/dev/./urandom  instead of the default 
> /dev/random, all problems go away.
Classic entropy problem.

> It's rather strange that multiple 7.3 RHEL systems are showing this 
> behavior very consistently. RHEL 7.2, 8.0 seem just fine in my 
> testing. No issues when u

Re: [OT] Weirdest Tomcat Behavior Ever?

[Christopher Schultz]

Eric,

On 11/25/20 09:34, Eric Robinson wrote:

-Original Message-
From: Christopher Schultz 
Sent: Tuesday, November 24, 2020 8:11 AM
To: users@tomcat.apache.org
Subject: Re: Weirdest Tomcat Behavior Ever?

Also, v5.0.8 is like 13 years old. Eric, you guys *really* have to upgrade that.
Somewhat surprisingly, there are CVEs against that library which allow
unauthenticated remote attackers to take-over the MySQL client
connections opened by that library.



Chris, I'm in full agreement with you on that. We'd love to update 
the connector but we are under vendor constraints. They only support

certain versions.


*eyeroll*

How about "the officially-supported version of the library causes 
ClassLoader leaks and leaves threads running when it shouldn't. Please 
support a non-ancient version of this library"?


You should be able to argue based solely upon security reports that them 
not supporting a newer version reduces the security of your system which 
is not acceptable. Read your SLAs to see if you can force them to do the 
right thing.


Honestly, dropping the newer version of the driver on top of the old one 
will work in nearly 100% of the cases. I have a lot of experience with 
the Connector/J library and I've never had an upgrade break anything.


Since I'm dumping on Connector/J right now, I just have to post this 
item from the changelog of the most recent 5.1.x release[1] because it's 
priceless:


"
Bugs Fixed

When trying to set a parameter for a PreparedStatement using the 
method PreparedStatement.setObject(parameterIndex, "false", 
Types.BOOLEAN), the value was set to true instead of false. (Bug 
#30911870, Bug #98237)

"

So, to recap: in MySQL Connector/J versions before 5.1.49, "false" means 
"true".


*sigh* This *is* year-2020, I guess.

What better reason to upgrade?

-chris

[1] https://dev.mysql.com/doc/relnotes/connector-j/5.1/en/news-5-1-49.html

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tomcat connection pool per database (postgres)

[Rob Sargent]

For those scoring at home, I'm going to try managing pool-per-user 
myself for the potential jmx gain and as I said, two will be amazing 
success.



On 11/25/20 9:40 AM, Christopher Schultz wrote:

Phil and Rob,

On 11/24/20 11:26, Phil Steitz wrote:

On 11/24/20 8:52 AM, Rob Sargent wrote:
Perhaps I read too much into the description of "The tomcat JDBC 
Connection Pool" page?


TheJDBC Connection Pool|org.apache.tomcat.jdbc.pool|is a replacement 
or an alternative to theApache Commons DBCP 
connection pool.


I reacted to the "replacement" bit. Are both equally sound, 
supported, surviving?


Yes.  I can't speak for jdbc-pool, but it looks like it is being 
actively maintained.


We really need to change the language on that page. It should really 
say something like "Back in the day, the Tomcat team (in retrospect) 
overreacted to the limitations of DBCP and decided to build a new 
high-performance pool. These days, the newer DBCP2 is great, and 
jdbc-pool also exists. They are two reasonably equivalent ways to 
solve the same problem. The default is DBCP and it's fine. Switching 
to jdbc-pool is also fine; it does have a few minor features that you 
can't (currently) get from jdbc-pool, and you will know right away if 
you actually need them."



The tomcat website text has never been updated to reflect this.


Sorry about that. The current text says "Note that this does not apply 
to Commons DBCP 2.x." in a few places. It's obviously very partisan 
when it doesn't need to be.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Weirdest Tomcat Behavior Ever?

[Christopher Schultz]

Mark,

On 11/24/20 09:56, Mark Thomas wrote:

On 24/11/2020 14:11, Christopher Schultz wrote:

On 11/20/20 11:08, Mark Thomas wrote:





A second look at the strace file from yesterday provided hard evidence
of a native library mis-using file descriptors and strong circumstantial
evidence linking that same library to a specific instance of the
observed failure.


Interesting. How can you tell it's a library instead of, for example,
the JVM itself (which is of course itself a set of native libraries).


strace shows the .so file being accessed, then we see the process
forking, a pipe set up between the parent and child and the child
calling execve on the same .so file. After a normal clean-up the parent
then calls close on the two file descriptors associated with the pipe
for a second time.


Looks pretty straightforward. Clearly not Connector/J... at least not 
one with a normal configuration.



I'm as sure as I can be without access to the source code for the .so
file that it is mis-handling file descriptors.


I'm assuming that when you say "native library" you mean "some native
component which is not a part of the core JVM".


The .so file in question is not part of the JVM. It appears to be a
third-party native library that ships as part of the commercial web
application where the original issue is observed.


Aha!


TL;DR, an issue in an external library, not a Tomcat issue.





So does this really look like a (pretty serious) bug in a native
library? Any idea which one?


I'm reasonably sure but I had to make a couple of assumptions based on
file paths to ID the library. I've passed that info to Eric but until it
is confirmed it doesn't seem right to name it on list.


+1 and +1

Reporting this privately to the vendor (by Eric) seems appropriate to me.

Strange that the library is getting a direct reference to the response's 
file descriptor. That seems (to me) to be a tactical mistake in an 
implementation. Then again, I'm not looking at what is actually being 
accomplished, there... it may make perfect sense in context.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: tomcat connection pool per database (postgres)

[Christopher Schultz]

Phil and Rob,

On 11/24/20 11:26, Phil Steitz wrote:

On 11/24/20 8:52 AM, Rob Sargent wrote:
Perhaps I read too much into the description of "The tomcat JDBC 
Connection Pool" page?


TheJDBC Connection Pool|org.apache.tomcat.jdbc.pool|is a replacement 
or an alternative to theApache Commons DBCP 
connection pool.


I reacted to the "replacement" bit. Are both equally sound, supported, 
surviving?


Yes.  I can't speak for jdbc-pool, but it looks like it is being 
actively maintained.


We really need to change the language on that page. It should really say 
something like "Back in the day, the Tomcat team (in retrospect) 
overreacted to the limitations of DBCP and decided to build a new 
high-performance pool. These days, the newer DBCP2 is great, and 
jdbc-pool also exists. They are two reasonably equivalent ways to solve 
the same problem. The default is DBCP and it's fine. Switching to 
jdbc-pool is also fine; it does have a few minor features that you can't 
(currently) get from jdbc-pool, and you will know right away if you 
actually need them."


The tomcat website text has never been 
updated to reflect this.


Sorry about that. The current text says "Note that this does not apply 
to Commons DBCP 2.x." in a few places. It's obviously very partisan when 
it doesn't need to be.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Bouncy Castle FIPS on RHEL 7.3

[Christopher Schultz]

Amit,

On 11/24/20 11:21, Amit Pande wrote:

Probably not directly related to Tomcat but still sharing. Advanced
apologies for that.

I am using bouncy castle FIPS library and observed that specifically
on RHEL 7.3, the library usage is causing tremendous slowness.
Note that BC is pure-Java, so you should pretty much expect "tremendous 
slowness". The reason that the Sun JCE/JSSE provider is (or can be) very 
fast is because it's using both native code (which is only marginally 
faster than well-JIT'ed Java bytecode, if at all) and also hardware 
support when it's available. There is no way that I know of for BC's 
pure-Java crypto implementations to use hardware acceleration. AES on 
silicon is *much* faster than AES in software.



e.g. below key tool command taking several minutes to finish.


Uhh... that's not software versus hardware. You might expect an order of 
magnitude difference between BC and hardware-accelerated SunJCE. But 
minutes to generate a 2048-bit RSA key? Something is wrong.



keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass 
org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair  -keyalg RSA -keypass "Test123" 
-validity 3650 -dname "CN=Test_HostName, ou=My Org Unit, o=My Org" -storepass "Test123" -keystore 
"/tmp/test_bc.bcfks" -storetype BCFKS -v -alias test_entry

However, when I add the JVM option
-J-Djava.security.egd=file:/dev/./urandom  instead of the default
/dev/random, all problems go away.

Classic entropy problem.

It's rather strange that multiple 7.3 RHEL systems are showing this 
behavior very consistently. RHEL 7.2, 8.0 seem just fine in my

testing. No issues when using /dev/random along with JKS type key
stores (including RHEL 7.3).

Even Tomcat is flagging the slowness to generate the secure random
numbers.

"17-Nov-2020 19:24:59.142 WARNING [Catalina-utility-2] 
org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of 
SecureRandom instance for session ID generation using [SHA1PRNG] took [68,745] 
milliseconds."

Apparently, according to FIPS experts we should not use /dev/./urandom.


Correct. FIPS assumes that 100% of your cryptographic operations are of 
the *utmost highly-classified state-secret-style nuclear-launch-code 
flashing-purple* security level.


Sometimes you just don't need to have perfect entropy.

For long-lived keys, really good entropy is a Good Thing. So to generate 
your RSA keys, I'd say go ahead and use the "high quality entropy 
source". But for randomness for session id generation, and for 
temporary, bulk encryption TLS session keys, /dev/urandom is perfectly fine.


So, while I continue to explore further, sharing this in case any 
one has seen this behavior or be aware of such potential issue as it

may tie in to their Tomcat (or web apps to be deployed in Tomcat).
You might be able to justify using /dev/urandom (which must be spelled 
/dev/./urandom to force Java to actually use it) in your web 
application, but I would go ahead and let keytool take as long as it needs.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Weirdest Tomcat Behavior Ever?

[Eric Robinson]

> -Original Message-
> From: Mark Thomas 
> Sent: Tuesday, November 24, 2020 8:57 AM
> To: users@tomcat.apache.org
> Subject: Re: Weirdest Tomcat Behavior Ever?
>
> On 24/11/2020 14:11, Christopher Schultz wrote:
> > On 11/20/20 11:08, Mark Thomas wrote:
>
> 
>
> >> A second look at the strace file from yesterday provided hard
> >> evidence of a native library mis-using file descriptors and strong
> >> circumstantial evidence linking that same library to a specific
> >> instance of the observed failure.
> >
> > Interesting. How can you tell it's a library instead of, for example,
> > the JVM itself (which is of course itself a set of native libraries).
>
> strace shows the .so file being accessed, then we see the process forking, a
> pipe set up between the parent and child and the child calling execve on the
> same .so file. After a normal clean-up the parent then calls close on the two
> file descriptors associated with the pipe for a second time.
>
> I'm as sure as I can be without access to the source code for the .so file 
> that it
> is mis-handling file descriptors.
>
> > I'm assuming that when you say "native library" you mean "some native
> > component which is not a part of the core JVM".
>
> The .so file in question is not part of the JVM. It appears to be a 
> third-party
> native library that ships as part of the commercial web application where the
> original issue is observed.
>
> >> TL;DR, an issue in an external library, not a Tomcat issue.
>
> 
>
> > So does this really look like a (pretty serious) bug in a native
> > library? Any idea which one?
>
> I'm reasonably sure but I had to make a couple of assumptions based on file
> paths to ID the library. I've passed that info to Eric but until it is 
> confirmed it
> doesn't seem right to name it on list.
>
> Mark
>

The full evidence package was submitted to the application vendor this morning, 
including all relevant logs, packet captures, strace files, and the 
accompanying analysis (courtesy of Mark) which seems pretty conclusive. We're 
eager to hear their feedback. That said, I'm not too shy about mentioning the 
names of the suspected libraries, as long as we're clear that the cause is not 
confirmed, and specific about the evidence that points in that direction. I may 
do so in a follow-up posting. After all, we already know the names of the other 
major components involved--tomcat, java, and the jdbc connector. Naming the 
suspect libraries may do someone else a service who has a gut feeling about 
them but hasn't seen solid evidence to support their concerns.

-Eric

> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Weirdest Tomcat Behavior Ever?

[Eric Robinson]

> -Original Message-
> From: Christopher Schultz 
> Sent: Tuesday, November 24, 2020 8:11 AM
> To: users@tomcat.apache.org
> Subject: Re: Weirdest Tomcat Behavior Ever?
>
> Mark,
>
> On 11/20/20 11:08, Mark Thomas wrote:
> > On 20/11/2020 15:43, Eric Robinson wrote:
> >>> From: Mark Thomas 
> >
> >
> > 
> >
> >>> This information might turn out to be unnecessary. I have been
> >>> re-looking at the previous logs and I think I have enough evidence
> >>> to ID a root cause. I'll email off-list as I need to quote logs to prove 
> >>> my
> point.
> >>>
> >>
> >> I'll be watching my inbox like a hawk!
> >
> > Sent.
> >
> > For the curious, the analysis indicated either JVM code or a native
> > library was incorrectly closing the file descriptor associated with
> > the socket being used by an in progress HTTP request.
>
> FWIW, Connector/J is a Type 4 JDBC Driver (aka "pure Java"), so no native
> components.
>
> I'm not sure how it would be able to close the connection.
>
> Also, v5.0.8 is like 13 years old. Eric, you guys *really* have to upgrade 
> that.
> Somewhat surprisingly, there are CVEs against that library which allow
> unauthenticated remote attackers to take-over the MySQL client
> connections opened by that library.
>

Chris, I'm in full agreement with you on that. We'd love to update the 
connector but we are under vendor constraints. They only support certain 
versions.

> > A second look at the strace file from yesterday provided hard evidence
> > of a native library mis-using file descriptors and strong
> > circumstantial evidence linking that same library to a specific
> > instance of the observed failure.
>
> Interesting. How can you tell it's a library instead of, for example, the JVM
> itself (which is of course itself a set of native libraries).
> I'm assiming that when you say "native library" you mean "some native
> component which is not a part of the core JVM".
>
> > TL;DR, an issue in an external library, not a Tomcat issue.
>
> I've recently been trying to optimize IO throughput of certain web-to-
> database (and vice-versa) operations in my own web applications.
> The gist is that we are wrapping request InputStreams or JDBC InputStreams
> (depending upon whether we are reading from request->db or
> db->response) to allow streaming directly in or out of the database with
> a minimum of buffering.
>
> If such a scheme were to be implemented particularly badly (e.g.
> allowing the database code direct-access to the request and response
> objects instead of just the stream(s)), it would certainly be possible
> for such an implementation to close the response's output stream before
> the servlet was expecting it.
>
> But such an implementation would be caught red-handed if a simple
> wrapper around the response stream were to be installed and simply log
> all the calls to close(), which I think was one of your first debugging
> steps, here.
>
> So does this really look like a (pretty serious) bug in a native
> library? Any idea which one?
>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org

Disclaimer : This email and any files transmitted with it are confidential and 
intended solely for intended recipients. If you are not the named addressee you 
should not disseminate, distribute, copy or alter this email. Any views or 
opinions presented in this email are solely those of the author and might not 
represent those of Physician Select Management. Warning: Although Physician 
Select Management has taken reasonable precautions to ensure no viruses are 
present in this email, the company cannot accept responsibility for any loss or 
damage arising from the use of this email or attachments.