Migrating from TC9 to TC10. Can the jakartaConverter log problem cases?

[Martin Knoblauch]

Hi,

 I am in the process of migrating a large application from Tomcat9 to
Tomcat10. While we have fixed the javax->jakarta issues in our code
directly, we still need to specify jakartaConverter=TOMCAT to make the
application work correctly.

My question is now: does the converter log its work somewhere? How to
enable it? Sorry if this is a stupid question.

Thanks in advance
Martin
-- 
--
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de

How to setup Apache web server for a Tomcat deployed Spring application

[Martin Moore]

Hello,

I have a situation where I want to call an Tomcat deployed Spring
application remotely without adding the port number (8080), I had tried to
use 80 in Connector but wasn't able to connect to it when outside the LAN.
So I resorted to creating a proxy server using Apache2 Web Server (along
side the Tomcat application server).

Unfortunately when I call the app using name1.domain.com/app/login it times
out and fails.
Following are the configuration for Apache2 and Tomcat:
In server.xml (Tomcat V8)
http://qadat.qfls.idealab.unical.it/>com"
   proxyPort="80"
  />

httpd.conf (under conf/ in Apache2)
...
LoadModule proxy_module modules/mod_proxy.so
...
ServerName  name1.domain. com:80


   ProxyPass /app/login http://localhost:8080/app/login

   ProxyPassReverse /app/login http://localhost:8080/app/login



To note that on the local machine tomcat returns the app through
http://localhost:8080/app/login 

How to make the app requests proxied so that name1.domain.
com/app/login works and calls the
localhost:8080/app/login to return the Tomcat Spring app

To note that DocumentRoot was added and then removed and that the app
resides in webapps/ROOT

Thanks.

Re: [mod_jk] Is it possible to set the value of a specific attribute via HTTP?

[Martin Knoblauch]

On 6/29/23 17:06, Rainer Jung wrote:
>
> > Since I try to push people into mod_proxy, I am hesitant to implement
> > more and more features which keep people from switching ;)
>
>
Hi Rainer,

 so, what do you suggest for the mod_jk retirement?
mod_proxy+mod_proxy_ajp, or just proxy to the HTTPS port? Is there an
equivalent to jkmanager with mod_proxy?

Thanks
Martin
-- 
------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de

CVE-2021-43980 completely fixed?

[Martin Garbe]

Hello all,

we use tomcat 9.0.62 in our environment and most likely hit the bug from 
CVE-2021-43980 (which should be fixed in this version).

Why do we think that we hit this bug?

  *
Since we refactored some e2e tests one test regularly fails because the client 
receives packets that should never be received by this client. The received 
packets belong to another TCP connection.
  *
We did a wireshark dump and can confirm that the packets were sent by tomcat. 
For a period of 4msec some packets are routed "into the wrong TCP connection" 
by tomcat.
  *
The bug seems to be triggered by some special timing + parallel processing 
situation.
  *
In the first step we assumed to hit a bug in AWS environment or TCP/IP stack 
but then found the bug report for tomcat.
  *   We used tomcat 9.0.60 when hitting the bug. By upgrading to 9.0.62 the 
bug kept appearing. Even 9.0.70 the bug exists.
  *
We replaced tomcat with undertow and all e2e tests run fine.

>From our point of view, we have an environment/configuration which triggers 
>this bug very often 100%).
If you need anybody to test further fixes, then we can help you. Please let us 
know.

Best regards,
Martin Garbe

Re: About granting permissions to Tomcat JVM

[Martin Moore]

@Rob  this is the command:   %CATALINA_HOME%\bin\startup.bat


Le dim. 9 oct. 2022 à 21:30, Rob Sargent  a écrit :

>
>
> > On Oct 9, 2022, at 8:54 AM, Martin Moore 
> wrote:
> >
> > @thomas.hoffm...@speed4trade.com.invalid
> >   how to use the debugger in
> this
> > case ?any documentation pls ?
> > @Rob Sargent i posted most of the methods used in the app, but yes JVM is
> > holding off on deleting the file...
>
> I think this might be a config issue. What is the full Java command with
> variables expanded
>
> >
> >> Le dim. 9 oct. 2022 à 17:31, Thomas Hoffmann (Speed4Trade GmbH)
> >>  a écrit :
> >>
> >> Hello,
> >>
> >>> About the methods i use in the application along with file.delete()
> >> (which is
> >>> what this post is about) I use File[] files = new
> >>> File("C:\\SomeFolder").listFiles();
> >>> file.getName();
> >>> file.lastModified()
> >>> Path path = Paths.get(fileName);
> >>> Would any of these cause the lock on the file ?
> >>
> >> They don't look suspicious to me. What you could do is, use the debugger
> >> to go step by step.
> >> In parallel, use the ProcessExplorer to see at which point the file
> handle
> >> appears.
> >> Maybe it is more hidden in some other routines which might use a
> >> MappedByteBuffer to read or write.
> >>
> >>
> >>>
> >>> Le dim. 9 oct. 2022 à 16:03, Thomas Hoffmann (Speed4Trade GmbH)
> >>>  a écrit :
> >>>
> >>>>
> >>>>> -Ursprüngliche Nachricht-
> >>>>> Von: Martin Moore 
> >>>>> Gesendet: Sonntag, 9. Oktober 2022 15:37
> >>>>> An: thomas.hoffm...@speed4trade.com.invalid; Tomcat Users List
> >>>>> 
> >>>>> Betreff: Re: About granting permissions to Tomcat JVM
> >>>>>
> >>>>> the ProcessExplorer shows that a Java process is running on the file
> >>>>> and
> >>>> this
> >>>>> only after actually performing the delete from Java.
> >>>>
> >>>>
> >>>> Some possibilities for file locking.
> >>>> - the upload process didn’t close the streams properly
> >>>> - You use http2 and sendfile
> >>>> - You are using a MappedByteBuffer before deletion:
> >>>> https://bugs.openjdk.org/browse/JDK-4724038
> >>>>
> >>>> ... just some ideas which cause file locking ...
> >>>>
> >>>>> Le dim. 9 oct. 2022 à 15:23, Thomas Hoffmann (Speed4Trade GmbH)
> >>>>>  a écrit :
> >>>>>
> >>>>>> Hello,
> >>>>>> this might be a behavior of the underlying OS.
> >>>>>> If the file is locked, it is marked for deletion and when the file
> >>>>>> lock is released, the file is physically deleted.
> >>>>>>
> >>>>>> Maybe you can check with ProcessExplorer from MS whether there is
> >>>>>> an open file handle on this file.
> >>>>>>
> >>>>>> Greetings,
> >>>>>> Thomas
> >>>>>>
> >>>>>>> -Ursprüngliche Nachricht-
> >>>>>>> Von: Martin Moore 
> >>>>>>> Gesendet: Sonntag, 9. Oktober 2022 09:56
> >>>>>>> An: Tomcat Users List ;
> >>>>>>> ma...@apache.org
> >>>>>>> Betreff: Re: About granting permissions to Tomcat JVM
> >>>>>>>
> >>>>>>> Hello Mark,
> >>>>>>>
> >>>>>>> I don't know if the SecurityManager is enabled or not (how to
> >>>>>>> disable it
> >>>>>>> btw?)
> >>>>>>> I set the env var CATALINA_HOME to be C:/Program
> >>>>>>> Files/Apache-Tomcat-8/ The files in question are stored in
> >>>>>>> Desktop/SomeFolder
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> Le dim. 9 oct. 2022 à 08:00, Mark Thomas  a
> >>>>>>> écrit
> >>>> :
> >>>>>>>
> >>>>>>>> On 08/10/2022 17:36, Martin Moore wrote:
> >>>>>>>>> Hello,
> >>>>>

Re: About granting permissions to Tomcat JVM

[Martin Moore]

@thomas.hoffm...@speed4trade.com.invalid
  how to use the debugger in this
case ?any documentation pls ?
@Rob Sargent i posted most of the methods used in the app, but yes JVM is
holding off on deleting the file...

Le dim. 9 oct. 2022 à 17:31, Thomas Hoffmann (Speed4Trade GmbH)
 a écrit :

> Hello,
>
> > About the methods i use in the application along with file.delete()
> (which is
> > what this post is about) I use File[] files = new
> > File("C:\\SomeFolder").listFiles();
> > file.getName();
> > file.lastModified()
> > Path path = Paths.get(fileName);
> > Would any of these cause the lock on the file ?
>
> They don't look suspicious to me. What you could do is, use the debugger
> to go step by step.
> In parallel, use the ProcessExplorer to see at which point the file handle
> appears.
> Maybe it is more hidden in some other routines which might use a
> MappedByteBuffer to read or write.
>
>
> >
> > Le dim. 9 oct. 2022 à 16:03, Thomas Hoffmann (Speed4Trade GmbH)
> >  a écrit :
> >
> > >
> > > > -Ursprüngliche Nachricht-
> > > > Von: Martin Moore 
> > > > Gesendet: Sonntag, 9. Oktober 2022 15:37
> > > > An: thomas.hoffm...@speed4trade.com.invalid; Tomcat Users List
> > > > 
> > > > Betreff: Re: About granting permissions to Tomcat JVM
> > > >
> > > > the ProcessExplorer shows that a Java process is running on the file
> > > > and
> > > this
> > > > only after actually performing the delete from Java.
> > >
> > >
> > > Some possibilities for file locking.
> > > - the upload process didn’t close the streams properly
> > > - You use http2 and sendfile
> > > - You are using a MappedByteBuffer before deletion:
> > > https://bugs.openjdk.org/browse/JDK-4724038
> > >
> > > ... just some ideas which cause file locking ...
> > >
> > > > Le dim. 9 oct. 2022 à 15:23, Thomas Hoffmann (Speed4Trade GmbH)
> > > >  a écrit :
> > > >
> > > > > Hello,
> > > > > this might be a behavior of the underlying OS.
> > > > > If the file is locked, it is marked for deletion and when the file
> > > > > lock is released, the file is physically deleted.
> > > > >
> > > > > Maybe you can check with ProcessExplorer from MS whether there is
> > > > > an open file handle on this file.
> > > > >
> > > > > Greetings,
> > > > > Thomas
> > > > >
> > > > > > -Ursprüngliche Nachricht-
> > > > > > Von: Martin Moore 
> > > > > > Gesendet: Sonntag, 9. Oktober 2022 09:56
> > > > > > An: Tomcat Users List ;
> > > > > > ma...@apache.org
> > > > > > Betreff: Re: About granting permissions to Tomcat JVM
> > > > > >
> > > > > > Hello Mark,
> > > > > >
> > > > > > I don't know if the SecurityManager is enabled or not (how to
> > > > > > disable it
> > > > > > btw?)
> > > > > > I set the env var CATALINA_HOME to be C:/Program
> > > > > > Files/Apache-Tomcat-8/ The files in question are stored in
> > > > > > Desktop/SomeFolder
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Le dim. 9 oct. 2022 à 08:00, Mark Thomas  a
> > > > > > écrit
> > > :
> > > > > >
> > > > > > > On 08/10/2022 17:36, Martin Moore wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > I am facing a problem using Tomcat V8 with my J2ee app that
> > > > > > > > deletes
> > > > > > > (using
> > > > > > > > file.delete() Java 8) a file from disk (Windows). The file
> > > > > > > > is actually deleting only on application level meaning that
> > > > > > > > the application does not see the file anymore but if i open
> > > > > > > > the folder i still see the file which
> > > > > > > is
> > > > > > > > then locked by Java process. I only get the file to be
> > > > > > > > removed physically when i close the Tomcat instance.
> > > > > > > >
> > > > > > > > Does this problem relate to permissions in catalin

Re: About granting permissions to Tomcat JVM

[Martin Moore]

About the methods i use in the application along with file.delete() (which
is what this post is about)
I use File[] files = new File("C:\\SomeFolder").listFiles();
file.getName();
file.lastModified()
Path path = Paths.get(fileName);
Would any of these cause the lock on the file ?

Le dim. 9 oct. 2022 à 16:03, Thomas Hoffmann (Speed4Trade GmbH)
 a écrit :

>
> > -Ursprüngliche Nachricht-----
> > Von: Martin Moore 
> > Gesendet: Sonntag, 9. Oktober 2022 15:37
> > An: thomas.hoffm...@speed4trade.com.invalid; Tomcat Users List
> > 
> > Betreff: Re: About granting permissions to Tomcat JVM
> >
> > the ProcessExplorer shows that a Java process is running on the file and
> this
> > only after actually performing the delete from Java.
>
>
> Some possibilities for file locking.
> - the upload process didn’t close the streams properly
> - You use http2 and sendfile
> - You are using a MappedByteBuffer before deletion:
> https://bugs.openjdk.org/browse/JDK-4724038
>
> ... just some ideas which cause file locking ...
>
> > Le dim. 9 oct. 2022 à 15:23, Thomas Hoffmann (Speed4Trade GmbH)
> >  a écrit :
> >
> > > Hello,
> > > this might be a behavior of the underlying OS.
> > > If the file is locked, it is marked for deletion and when the file
> > > lock is released, the file is physically deleted.
> > >
> > > Maybe you can check with ProcessExplorer from MS whether there is an
> > > open file handle on this file.
> > >
> > > Greetings,
> > > Thomas
> > >
> > > > -Ursprüngliche Nachricht-
> > > > Von: Martin Moore 
> > > > Gesendet: Sonntag, 9. Oktober 2022 09:56
> > > > An: Tomcat Users List ; ma...@apache.org
> > > > Betreff: Re: About granting permissions to Tomcat JVM
> > > >
> > > > Hello Mark,
> > > >
> > > > I don't know if the SecurityManager is enabled or not (how to
> > > > disable it
> > > > btw?)
> > > > I set the env var CATALINA_HOME to be C:/Program
> > > > Files/Apache-Tomcat-8/ The files in question are stored in
> > > > Desktop/SomeFolder
> > > >
> > > > Thanks.
> > > >
> > > > Le dim. 9 oct. 2022 à 08:00, Mark Thomas  a écrit
> :
> > > >
> > > > > On 08/10/2022 17:36, Martin Moore wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I am facing a problem using Tomcat V8 with my J2ee app that
> > > > > > deletes
> > > > > (using
> > > > > > file.delete() Java 8) a file from disk (Windows). The file is
> > > > > > actually deleting only on application level meaning that the
> > > > > > application does not see the file anymore but if i open the
> > > > > > folder i still see the file which
> > > > > is
> > > > > > then locked by Java process. I only get the file to be removed
> > > > > > physically when i close the Tomcat instance.
> > > > > >
> > > > > > Does this problem relate to permissions in catalina.policy ?
> > > > >
> > > > > Unlikely.
> > > > >
> > > > > Are you using a SecurityManager?
> > > > >
> > > > > > How to solve this?
> > > > >
> > > > > Where, exactly, are you storing these files? Where, exactly, are
> > > > > CATALINA_HOME and CATALINA_BASE?
> > > > >
> > > > > Mark
> > > > >
> > > > > --
> > > > > --- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > > > For additional commands, e-mail: users-h...@tomcat.apache.org
> > > > >
> > > > >
> > >
> > > -
> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > For additional commands, e-mail: users-h...@tomcat.apache.org
> > >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: About granting permissions to Tomcat JVM

[Martin Moore]

the ProcessExplorer shows that a Java process is running on the file and
this only after actually performing the delete from Java.

Le dim. 9 oct. 2022 à 15:23, Thomas Hoffmann (Speed4Trade GmbH)
 a écrit :

> Hello,
> this might be a behavior of the underlying OS.
> If the file is locked, it is marked for deletion and when the file lock is
> released, the file is physically deleted.
>
> Maybe you can check with ProcessExplorer from MS whether there is an open
> file handle on this file.
>
> Greetings,
> Thomas
>
> > -Ursprüngliche Nachricht-
> > Von: Martin Moore 
> > Gesendet: Sonntag, 9. Oktober 2022 09:56
> > An: Tomcat Users List ; ma...@apache.org
> > Betreff: Re: About granting permissions to Tomcat JVM
> >
> > Hello Mark,
> >
> > I don't know if the SecurityManager is enabled or not (how to disable it
> > btw?)
> > I set the env var CATALINA_HOME to be C:/Program Files/Apache-Tomcat-8/
> > The files in question are stored in Desktop/SomeFolder
> >
> > Thanks.
> >
> > Le dim. 9 oct. 2022 à 08:00, Mark Thomas  a écrit :
> >
> > > On 08/10/2022 17:36, Martin Moore wrote:
> > > > Hello,
> > > >
> > > > I am facing a problem using Tomcat V8 with my J2ee app that deletes
> > > (using
> > > > file.delete() Java 8) a file from disk (Windows). The file is
> > > > actually deleting only on application level meaning that the
> > > > application does not see the file anymore but if i open the folder i
> > > > still see the file which
> > > is
> > > > then locked by Java process. I only get the file to be removed
> > > > physically when i close the Tomcat instance.
> > > >
> > > > Does this problem relate to permissions in catalina.policy ?
> > >
> > > Unlikely.
> > >
> > > Are you using a SecurityManager?
> > >
> > > > How to solve this?
> > >
> > > Where, exactly, are you storing these files? Where, exactly, are
> > > CATALINA_HOME and CATALINA_BASE?
> > >
> > > Mark
> > >
> > > -
> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > For additional commands, e-mail: users-h...@tomcat.apache.org
> > >
> > >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: About granting permissions to Tomcat JVM

[Martin Moore]

Hello Mark,

I don't know if the SecurityManager is enabled or not (how to disable it
btw?)
I set the env var CATALINA_HOME to be C:/Program Files/Apache-Tomcat-8/
The files in question are stored in Desktop/SomeFolder

Thanks.

Le dim. 9 oct. 2022 à 08:00, Mark Thomas  a écrit :

> On 08/10/2022 17:36, Martin Moore wrote:
> > Hello,
> >
> > I am facing a problem using Tomcat V8 with my J2ee app that deletes
> (using
> > file.delete() Java 8) a file from disk (Windows). The file is actually
> > deleting only on application level meaning that the application does not
> > see the file anymore but if i open the folder i still see the file which
> is
> > then locked by Java process. I only get the file to be removed physically
> > when i close the Tomcat instance.
> >
> > Does this problem relate to permissions in catalina.policy ?
>
> Unlikely.
>
> Are you using a SecurityManager?
>
> > How to solve this?
>
> Where, exactly, are you storing these files? Where, exactly, are
> CATALINA_HOME and CATALINA_BASE?
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

About granting permissions to Tomcat JVM

[Martin Moore]

Hello,

I am facing a problem using Tomcat V8 with my J2ee app that deletes (using
file.delete() Java 8) a file from disk (Windows). The file is actually
deleting only on application level meaning that the application does not
see the file anymore but if i open the folder i still see the file which is
then locked by Java process. I only get the file to be removed physically
when i close the Tomcat instance.

Does this problem relate to permissions in catalina.policy ?
How to solve this?

Re: Tomcat testing

[Martin Grigorov]

HI,

On Wed, Dec 8, 2021 at 11:20 AM Mark Thomas  wrote:

> On 08/12/2021 08:44, Hiran CHAUDHURI wrote:
> > Hello there.
> >
> > My organization requires that Tomcat releases - especially patches - get
> tested before we propagate them into production environments.
> > For sure similar tests are run by the ASF before releasing the software
> at all.
> >
> > Is there a way to run the very same tests within our organization? Where
> can I find out what automatic testing is already performed? Or would I have
> to develop a test suite from scratch?
>
> https://github.com/apache/tomcat/blob/main/BUILDING.txt


Here is what we do at Travis CI:
-
https://github.com/apache/tomcat/blob/2c5cea02157c3730ca498e0e26422dcaea5b8d4b/.travis.yml
-
https://github.com/apache/tomcat/blob/2c5cea02157c3730ca498e0e26422dcaea5b8d4b/.travis/antTest.sh


>
>
> Section 7.
>
> There is also the various CI systems:
> https://tomcat.apache.org/ci.html
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

RE: Tomcat 9.0.x JDBC connection pool does not always remove abandoned connections

[Martin, Gerhardt A]

Chris, 

really appreciate you taking some time to respond. See my replies inline below.

> -Original Message-
> From: Christopher Schultz 
> Sent: Thursday, October 14, 2021 12:19 PM
> To: users@tomcat.apache.org
> Subject: Re: Tomcat 9.0.x JDBC connection pool does not always remove
> abandoned connections
> 
> Gerhardt,
> 
> On 10/12/21 13:27, Martin, Gerhardt A wrote:
> > Running Tomcat 9.0.50 on Centos 7.9.x Linux and using Tomcat JDBC
> connection pool to connect to my application's databases. My app connects to
> about a dozen read only databases and one read/write database. Here is a
> typical resource definition with tuning configurations for the pool and the
> MSSQL server JDBC driver:
> >
> >  
> Note that these are the only relevant attributes:
> 
>  > factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
> 
> You are using tomcat-pool, not the default commons-dbcp.
 
100% aware of this and we plan to continue to use it.
> 
> > maxActive="20"
> > initialSize="1"
> > maxIdle="8"
> > minIdle="1"
> > maxWait="5000"
> > testOnBorrow="false"
> > testWhileIdle="false"
> > testOnConnect="false"
> > testOnReturn="false"
> > timeBetweenEvictionRunsMillis="1"
> > removeAbandonedTimeout="15"
> > removeAbandoned="true"
> > logAbandoned="false"
> > minEvictableIdleTimeMillis="90"
> > maxAge="60"
> 
> 
> 
> > Important NOTE: the framework we employ does NOT execute more than one
> > SQL statement per borrow from the connection pool. It's a constant
> > cycle of getConnection(borrow) ->execute 1
> > SQL->closeConnection(return)
> 
> That's kind of wasteful, but okay.
I agree to some extent, but we have had stable success with this inherited 
approach. Tough to change this in our current situation but I have had thoughts 
about it. 
> 
> > Why are we configured this way:
> > Ultimately we need to fail fast to avoid our APP going into death by stuck
> threads when we lose connectivity to the backend databases for a short period
> of time. ( for example a short 10 second period where the DB(s) is/are
> unreachable from the app servers).
> 
> Is this common?
 
Not common, but when it occurs it is very disruptive to our customer base, 
hence the need to improve resiliency.  
> 
> > -We want the evictor thread (Pool Cleaner) to run every 10 seconds -We
> > are only using the Pool Cleaner to clobber busy connections that have been
> running 15 seconds or more
> >NOTE: we rarely see SQL that takes longer than 2 seconds, most are
> > sub 100ms and the avg is about 4ms
> 
> Note that the connections aren't clobbered; they are simply dropped from the
> pool. The work they generated is still continuing...
> 
> > -We are not testing connections period -We want our connections
> > replaced with new fresh connections after 10 mins of use hence we set
> > maxAge=60 The connection pool settings work hand in hand with
> properties set on the SQL server JDBC driver in the URL:
> > The driver will kill any SQL that takes longer than 10 seconds
> 
> Ouch.
> 
> > -When everything else is OK, then the connection itself is returned
> > intact back to the connection pool -When the DB is unreachable, there
> > is another timer cancelQueryTimeout set to 10 seconds which is a safety 
> > valve
> for cases where the message to the DB server to cancel the SQL gracefully does
> not make it to the DB. After the cancleQueryTimeout, the client side of the
> driver gives up. This is problematic because it actually returns a bad 
> connection
> back to the connection pool.
> >
> > The combination of the connection pool settings to run the Pool cleaner 
> > every
> 10 seconds and removeAbandoned of 15 seconds should work together to
> remove any connection whose SQL execution time went past 10 seconds before
> the cancelQueryTimeout kicks in. This should prevent returning bad connections
> to the pool when bad things happen like a network switch going offline
> temporarily and dropping all packets between the app server hosting tomcat and
> the database server.
> 
> Sounds like you really just want to be validating connections, but you have
> explicitly disabled that feature. Why?
 
So we have a fairly high volume of SQL in our APPs and given our current 
borrow->execute SQL-> return situation, test on borrow or return doubles the 
number of SQL statements we run even if one of them is "SELECT 1". Our DBAs are 
not keen on this. Also there is

Tomcat 9.0.x JDBC connection pool does not always remove abandoned connections

[Martin, Gerhardt A]

Running Tomcat 9.0.50 on Centos 7.9.x Linux and using Tomcat JDBC connection 
pool to connect to my application's databases. My app connects to about a dozen 
read only databases and one read/write database. Here is a typical resource 
definition with tuning configurations for the pool and the MSSQL server JDBC 
driver:

 

Important NOTE: the framework we employ does NOT execute more than one SQL 
statement per borrow from the connection pool. It's a constant cycle of 
getConnection(borrow) ->execute 1 SQL->closeConnection(return) 


Why are we configured this way: 
Ultimately we need to fail fast to avoid our APP going into death by stuck 
threads when we lose connectivity to the backend databases for a short period 
of time. ( for example a short 10 second period where the DB(s) is/are 
unreachable from the app servers).   

-We want the evictor thread (Pool Cleaner) to run every 10 seconds
-We are only using the Pool Cleaner to clobber busy connections that have been 
running 15 seconds or more
  NOTE: we rarely see SQL that takes longer than 2 seconds, most are sub 100ms 
and the avg is about 4ms
-We are not testing connections period 
-We want our connections replaced with new fresh connections after 10 mins of 
use hence we set maxAge=60

The connection pool settings work hand in hand with properties set on the SQL 
server JDBC driver in the URL:
The driver will kill any SQL that takes longer than 10 seconds
-When everything else is OK, then the connection itself is returned intact back 
to the connection pool
-When the DB is unreachable, there is another timer cancelQueryTimeout set to 
10 seconds which is a safety valve for cases where the message to the DB server 
to cancel the SQL gracefully does not make it to the DB. After the
cancleQueryTimeout, the client side of the driver gives up. This is problematic 
because it actually returns a bad connection back to the connection pool. 

The combination of the connection pool settings to run the Pool cleaner every 
10 seconds and removeAbandoned of 15 seconds should work together to remove any 
connection whose SQL execution time went past 10 seconds before the 
cancelQueryTimeout kicks in. This should prevent returning bad connections to 
the pool when bad things happen like a network switch going offline temporarily 
and dropping all packets between the app server hosting tomcat and the database 
server. 

NOTE that for the MSSQL server driver all their timeouts default to wait 
indefinitely. This is one of the prime reasons why even short outages can kill 
our APP with stuck threads. Our Web app hits tomcat about 2000 requests per 
minute and each request is going to fire on AVG ~6 SQL statements so we are 
running about 12K SQL hits per minute. If we lose connectivity to a database 
for even 15 seconds without a means to fail fast we could easily overcome the 
allotted number of threads which will create a "log jam" that can take a very 
long time to recover.  

So I am very sorry for all this lenghty background but I thought it might help 
for readers and potential responders to know a bit about why we do what we do 
in our connection pool.

Finally now onto the problem: 

I was running a load test and simulating the temporary loss of connectivity to 
one or more of the databases used in the APP and I noticed a strange behavior 
that does not align with my understanding of how Tomcat JDBC removeAbandoned 
works. While most of my database connections coming from the pool did die 
shortly after the simulation started and "failed fast" there were a few 
connections that actually ran for up to 5 minutes before they terminated 
themselves. Given the configuration I explained I would have expected no 
database connection that was interacting with the database to live beyond 30 
seconds. ( giving that there could be some forward/reverse lag in the evictor 
thread timing where there could be a slight chance that the first run of the 
evictor thread might not catch every connection that was at the 15 seconds 
mark.) 

What I believe I saw in some cases was that the socket timeout that I set on 
the SQLServer JDBC driver for  300,000ms (5 mins) kicked in and read timeout 
was generated in the driver. This lead to a bad connection being returned to 
the connection pool and caused errors until the connection(s) aged out due to 
maxAge. 


Does anyone know why the Pool Cleaner would be able to run and NOT forcefully 
remove a connection from the tomcat JDBC pool that has been running more than 
configured removeAbandonedTimeout? 
I searched google and this list on MARC for any clues and I found nothing. I 
also reviewed the source code for the Tomcat JDBC module and was unable to see 
a place in the code that indicated it would not be able to clobber a running 
connection  that was borrowed from the pool.

Anyone able to help me understand what I saw in my testing? Could this be a 
Tomcat connection pool bug?


-Rega

Re: Enhancement: Additional user attributes queried by (some) realms

[Martin Grigorov]

On Tue, Jun 1, 2021 at 12:46 PM Carsten Klein  wrote:

>
>
> On 01/06/2021 10:18, Mark Thomas wrote:
>
> > I don't know if you can. I suspect not. By all means see if you can. I'm
> > mildly curious to find out the answer. Whether you can or not, you don't
> > need to.
>
> I found nothing to re-trigger the Travis CI build so far. However, now
> the CI test is 'successful'. Did you do anything about it?
>

I've restarted it few hours ago.
Only members of github.com/apache/tomcat team can restart builds via the
TravisCI UI. And most probably Apache Infra team members.


>
> Carsten
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Emdbedded Tomcat Issues

[Martin Grigorov]

On Mon, Apr 26, 2021 at 4:18 AM dana whitelow 
wrote:

>
> Embedded Tomcat 9.0.0.M6
>

Please use newer version. 9.0.0.M6 is released May 2016.
Latest 9.x release is 9.0.45.


>
>
>   Issue 1
>   //ctx.setAltDDName(
> "/home/user/microservices/AccountingApp/src/main/webapp/WEB-INF/web.xml");
> ctx.getServletContext().setAttribute(Globals.ALT_DD_ATTR,
> "/home/user/microservices/AccountingApp/src/main/webapp/WEB-INF/web.xml");
>
> Neither of the above  programmatic settings seems to have any
> effect on the message INFO: No global web.xml found
>
>
> INFO: Starting Servlet Engine: Apache Tomcat/9.0.0.M6
> Apr 26, 2021 1:58:59 AM org.apache.catalina.startup.ContextConfig
> getDefaultWebXmlFragment
> INFO: No global web.xml found
> Apr 26, 2021 1:58:59 AM org.omnifaces.ApplicationInitializer
> logOmniFacesVersion
> INFO: Using OmniFaces version 3.6.1
> Apr 26, 2021 1:58:59 AM org.apache.catalina.core.ApplicationContext log
> INFO: Marking servlet Faces Servlet as unavailable
> Apr 26, 2021 1:58:59 AM org.apache.catalina.core.StandardContext
> loadOnStartup
> SEVERE: Servlet [Faces Servlet] in web application [] threw load()
> exception
> java.util.MissingResourceException: Can't find javax.faces.LogStrings
> bundle from
>
> Issue 2
>
> from error message I think tomcat is looking for the file
> WebMessages.properties and WebMessages_es.properties from the
> faces-config.xml
> which I have in the src/main/resources/hacc/web/ and  in  directory
> src/main/resources/META-INF/resources/hacc/web ,
> but still it is giving me the error
> SEVERE: Servlet [Faces Servlet] in web application [] threw load()
> exception
> java.util.MissingResourceException: Can't find javax.faces.LogStrings
> bundle from
>
> at
> java.logging/java.util.logging.Logger.setupResourceInfo(Logger.java:2289)
> at java.logging/java.util.logging.Logger.(Logger.java:565)
> at
> java.logging/java.util.logging.LogManager.demandLogger(LogManager.java:522)
>
> 

Re: Memory vs. Heap Space

[Martin Knoblauch]

Hi Jerry,

 one thing that could happen is that for some reason your server machine
ran out of memory and in an attempt to keep the OS running it just killed
the biggest memory user which just might be your Tomcat process. We had
that happen to some development systems where the Tomcat "vanished" after 3
month uptime, because it was killed from the outside due to memory pressure.

But your request does not provide enough info to tell. What I would do is
turn on Java GC logging and see how the activity goes. Or use something
like "visualvm". Add "-XX:+HeapDumpOnOutOfMemoryError" to the Java options.
Look at the dump to see whether you have a leak in some rarely exercised
code path.

Cheers
Martin

On Tue, Apr 13, 2021 at 1:12 AM Jerry Malcolm 
wrote:

> I'm getting TC crashes (ver 8.5) with "out of memory" errors.  I
> understand heap space and the parameters to set it.  But I need a bit of
> help with "out of memory", not "out of heap space".   This is running in
> an EC2 on AWS.  It's a development server, so activity is minimal
> compared to production.  We also are running it on a smaller EC2 which
> is likely contributing to the problem. But interestingly the last crash
> happened in the middle of the night.  The last request came in 15
> minutes prior to the crash, and it was some php intrusion attempt URL
> that returned 404. Crashes don't happen often.  A couple of servers have
> failed once or twice a week.  And this is the first time the dev server
> crashed.  But I still really need to get a handle on it.
>
> We have a pretty large max heap space configured.  Does this problem
> have something to do with specifying a heap space larger than TC can
> allocate with physical memory?  I figured there was still the option of
> virtual memory and swapping if it ran out of physical memory.
>
> I need to learn more and tuning the memory usage.  Is there some formula
> that tells me the max heap space given a certain size of physical
> memory?  Or is there something else I'm totally missing about memory
> configuration?
>
> Thanks.
>
> Jerry
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

-- 
--
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de

Re: Tomcat 9.0

[Martin Grigorov]

Hi,

On Thu, Apr 8, 2021 at 3:25 AM Mohamed Eliyas Abdul Kadar <
mohamed.ka...@neogenomics.com> wrote:

> Hi All
> I am planning to use Tomcat for my development server. Initially we
> planned to go with version Tomcat 9.0.41. Now I see newer versions are
> release on top of that and see the latest version is Tomcat 9.0.45. Please
> let me know if there is any major fix of Tomcat 9.0.41 made on higher
> versions or we are good with Tomcat 9.0.44 as Tomcat 9.0.45 is not having
> any release date.
>

Please consult with https://tomcat.apache.org/tomcat-9.0-doc/changelog.html
for the changes in each version.

Martin


>
> Regards
> Eliyas
> This communication and its attachments contain confidential information
> and is intended only for the named addressee. If you are not the named
> addressee you should not disseminate, distribute or copy this
> communication. Please notify the sender immediately if you have received
> this communication by mistake and delete or destroy this communication.
> Communications cannot be guaranteed to be secured or error-free as
> information could be intercepted, corrupted, lost, destroyed, arrive late
> or incomplete, or contain viruses. The sender therefore does not accept
> liability for any errors or omissions in the contents of this communication
> which arise as a result of transmission. If verification is required please
> request a hard-copy version. NeoGenomics Laboratories, 12701 Commonwealth
> Dr, Fort Myers, FL 33913, http://www.neogenomics.com (2021)
>

Re: Tomcat Error: VersionLoggerListener

[Martin Grigorov]

Hi,

On Thu, Apr 1, 2021 at 1:42 PM KISHOR S.JADHAV 
wrote:

> Hi Team,
>  I am trying to run my war file in tomcat version 8.5 is not working so I
> have decrease the tomcat version and using the Tomcat8  but in this, I am
> getting the below error,
>
> [image: image.png]
>
>
> [image: image.png]
>

Please copy/paste the text instead of creating screenshots.
Attachments are not allowed in the mailing lists.


>
>  Getting this error *org.apache.catalina.startup.VersionLoggerListener
> log*
>  I am not able to find the correct solution for this issue, Request you
> to can you please help me to solve this issue
>
> Thanks and Regards,
> Kishor Jadhav
>
>

Re: Wait... NULL address in java.net.BindException: Address already in use (Bind failed) ???

[Martin Grigorov]

On Wed, Mar 17, 2021, 20:27 Eric Robinson  wrote:

> > From: Martin Grigorov 
> > Sent: Wednesday, March 17, 2021 12:45 PM
> > To: Tomcat Users List 
> > Subject: Re: Wait... NULL address in java.net.BindException: Address
> already
> > in use (Bind failed)  ???
> >
> > Hi,
> >
> > On Wed, Mar 17, 2021, 19:34 Eric Robinson 
> > wrote:
> >
> > > Getting error:
> > >
> > > java.net.BindException: Address already in use (Bind failed)
> > > :3787
> > >
> >
> > Please paste more lines of the exception.
> > Also please tell us which version of JDK/JRE you use.
> > This exception is very cryptic and does not usually tell which address
> is in use.
> > I.e. 3787 is not the port, as you might think. Most probably it is a
> line in some
> > class.
> >
>
> Tomcat: Apache Tomcat/8.5.51
> JVM: 1.8.0_241-b08
>
> The following error appears in catalina.out under tomcat 8. It does not
> mention the null. We tried it under tomcat 7 as well, and that is where it
> mentions the null.
>
> 17-Mar-2021 11:12:54.039 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio-3787"]
> 17-Mar-2021 11:12:54.048 SEVERE [main]
> org.apache.catalina.core.StandardService.initInternal Failed to initialize
> connector [Connector[HTTP/1.1-3787]]
>

This line says that 3787 is the port indeed.
Are you sure it is not bound?


org.apache.catalina.LifecycleException: Protocol handler
> initialization failed
> at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:1032)
> at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
> org.apache.catalina.core.StandardService.initInternal(StandardService.java:552)
> at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
> org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:848)
> at
> org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at
> org.apache.catalina.startup.Catalina.load(Catalina.java:639)
> at
> org.apache.catalina.startup.Catalina.load(Catalina.java:662)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:303)
> at
> org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch
> .ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
> at sun.nio.ch
> .ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
> at org.apache.tomcat.util.net
> .NioEndpoint.bind(NioEndpoint.java:221)
> at org.apache.tomcat.util.net
> .AbstractEndpoint.init(AbstractEndpoint.java:1118)
> at org.apache.tomcat.util.net
> .AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:223)
> at
> org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:587)
> at
> org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74)
> at
> org.apache.catalina.connector.Connector.initInternal(Connector.java:1030)
> ... 13 more
>
>
> >
> > > I know how to fix the infamous "Address already in use (Bind failed)"
> > > problem when there is another process already listening on a port.
> > > However, I have confirmed with netstat and fuser that there is no
> > > other process listening on that port. Could the problem be that the
> > > host address is null for some reason? I don't recall seeing that
> > > before, and Google diving came up dry.
> > >
> > > -Eric
> > >
> > >
> > >
> > > Disclaimer : This email and any files transmitted with it are
> > > confidential and intended solely for intended recipients. If you are
> > > not the named addressee you should not disseminate, distribute, copy or
> > alter this email.
> > > A

Re: Wait... NULL address in java.net.BindException: Address already in use (Bind failed) ???

[Martin Grigorov]

Hi,

On Wed, Mar 17, 2021, 19:34 Eric Robinson  wrote:

> Getting error:
>
> java.net.BindException: Address already in use (Bind failed) :3787
>

Please paste more lines of the exception.
Also please tell us which version of JDK/JRE you use.
This exception is very cryptic and does not usually tell which address is
in use. I.e. 3787 is not the port, as you might think. Most probably it is
a line in some class.


> I know how to fix the infamous "Address already in use (Bind failed)"
> problem when there is another process already listening on a port. However,
> I have confirmed with netstat and fuser that there is no other process
> listening on that port. Could the problem be that the host address is null
> for some reason? I don't recall seeing that before, and Google diving came
> up dry.
>
> -Eric
>
>
>
> Disclaimer : This email and any files transmitted with it are confidential
> and intended solely for intended recipients. If you are not the named
> addressee you should not disseminate, distribute, copy or alter this email.
> Any views or opinions presented in this email are solely those of the
> author and might not represent those of Physician Select Management.
> Warning: Although Physician Select Management has taken reasonable
> precautions to ensure no viruses are present in this email, the company
> cannot accept responsibility for any loss or damage arising from the use of
> this email or attachments.
>

Re: Keeping a session active with ajax

[Martin Grigorov]

Chris,

On Mon, Mar 15, 2021, 18:28 Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Martin,
>
> On 3/15/21 04:08, Martin Grigorov wrote:
> > Hi Maxim,
> >
> > On Mon, Mar 15, 2021 at 9:46 AM Maxim Solodovnik 
> > wrote:
> >
> >> Maybe offtopic
> >> but websocket requests seems to not warming up the HttpSession
> >>
> >
> > The websocket messages do not bring the JSESSIONID cookie/request
> parameter
> > and there is no way to lookup the http session from the web container
> > (Tomcat).
> > Wicket creates a copy of the HttpSession at WebSocket connection creation
> > time and uses it until the closing of the connection.
> > So, yes, the real http session may expire if you use only websocket based
> > communication.
>
> Yes, but that's not AJAX, as the OP mentioned. Perhaps "AJAX" just means
> "communication using javascript without 'traditional' navigation".
>

Did you read my first answer and Maxim's email?


> Thanks,
> -chris
> >> Tomcat 9.0.44
> >> Wicket 9.2.0 :)
> >>
> >> On Mon, 15 Mar 2021 at 14:43, Martin Grigorov 
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Which version of Tomcat ?
> >>>
> >>> On Mon, Mar 15, 2021 at 8:25 AM Jerry Malcolm 
> >>> wrote:
> >>>
> >>>> In my app a logged in user will access a page.  But then all of the
> >>>> interaction on the page is done back and forth with ajax calls to the
> >>>> server.  The ajax activities within the page could last well over the
> >> 30
> >>>> minute session timeout. Even though the user has been actively working
> >>>> for 30-45 minutes with lots of server calls, all of a sudden they
> >> aren't
> >>>> logged in any more.  The ajax calls are definitely successfully
> >>>> accessing the session since the ajax APIs utilize the logged on user,
> >>>> roles, etc.  They just aren't pinging the keep-alive bit in the
> session
> >>>> for some reason.  I'm not really sure what would be different in the
> >>>> calls from a regular page call from the browser vs an ajax call.  They
> >>>> both make a request that goes to a servlet or jsp and they both get
> >>>> response data back from tomcat.  But the session times out if I don't
> >> do
> >>>> a real page call.  What am I missing?
> >>>>
> >>>
> >>> Nothing!
> >>> Any request that uses the HttpSession
> >>> (HttpServletRequest.getHttpSession(true)) will "touch" the session and
> >>> update its "lastUsed" time, so it should not time out during active
> usage
> >>> of the application.
> >>> There must be something else that invalidates the http session.
> >>>
> >>>
> >>>>
> >>>> I really don't want to change the 'inactivity' timeout.  I just need
> to
> >>>> figure out how to tell tomcat with all of the ajax calls that the
> >>>> session is still very active.  Is there an easy solution?
> >>>>
> >>>> Thx
> >>>>
> >>>> Jerry
> >>>>
> >>>> BTW... TC 8.5
> >>>>
> >>>>
> >>>> -
> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>>
> >>>>
> >>
> >>
> >>
> >> --
> >> Best regards,
> >> Maxim
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Keeping a session active with ajax

[Martin Grigorov]

Hi Maxim,

On Mon, Mar 15, 2021 at 9:46 AM Maxim Solodovnik 
wrote:

> Maybe offtopic
> but websocket requests seems to not warming up the HttpSession
>

The websocket messages do not bring the JSESSIONID cookie/request parameter
and there is no way to lookup the http session from the web container
(Tomcat).
Wicket creates a copy of the HttpSession at WebSocket connection creation
time and uses it until the closing of the connection.
So, yes, the real http session may expire if you use only websocket based
communication.

Martin

P.S. Congrats for Apache OpenMeetings 6.0!


>
> Tomcat 9.0.44
> Wicket 9.2.0 :)
>
> On Mon, 15 Mar 2021 at 14:43, Martin Grigorov 
> wrote:
> >
> > Hi,
> >
> > Which version of Tomcat ?
> >
> > On Mon, Mar 15, 2021 at 8:25 AM Jerry Malcolm 
> > wrote:
> >
> > > In my app a logged in user will access a page.  But then all of the
> > > interaction on the page is done back and forth with ajax calls to the
> > > server.  The ajax activities within the page could last well over the
> 30
> > > minute session timeout. Even though the user has been actively working
> > > for 30-45 minutes with lots of server calls, all of a sudden they
> aren't
> > > logged in any more.  The ajax calls are definitely successfully
> > > accessing the session since the ajax APIs utilize the logged on user,
> > > roles, etc.  They just aren't pinging the keep-alive bit in the session
> > > for some reason.  I'm not really sure what would be different in the
> > > calls from a regular page call from the browser vs an ajax call.  They
> > > both make a request that goes to a servlet or jsp and they both get
> > > response data back from tomcat.  But the session times out if I don't
> do
> > > a real page call.  What am I missing?
> > >
> >
> > Nothing!
> > Any request that uses the HttpSession
> > (HttpServletRequest.getHttpSession(true)) will "touch" the session and
> > update its "lastUsed" time, so it should not time out during active usage
> > of the application.
> > There must be something else that invalidates the http session.
> >
> >
> > >
> > > I really don't want to change the 'inactivity' timeout.  I just need to
> > > figure out how to tell tomcat with all of the ajax calls that the
> > > session is still very active.  Is there an easy solution?
> > >
> > > Thx
> > >
> > > Jerry
> > >
> > > BTW... TC 8.5
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > > For additional commands, e-mail: users-h...@tomcat.apache.org
> > >
> > >
>
>
>
> --
> Best regards,
> Maxim
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Keeping a session active with ajax

[Martin Grigorov]

Hi,

Which version of Tomcat ?

On Mon, Mar 15, 2021 at 8:25 AM Jerry Malcolm 
wrote:

> In my app a logged in user will access a page.  But then all of the
> interaction on the page is done back and forth with ajax calls to the
> server.  The ajax activities within the page could last well over the 30
> minute session timeout. Even though the user has been actively working
> for 30-45 minutes with lots of server calls, all of a sudden they aren't
> logged in any more.  The ajax calls are definitely successfully
> accessing the session since the ajax APIs utilize the logged on user,
> roles, etc.  They just aren't pinging the keep-alive bit in the session
> for some reason.  I'm not really sure what would be different in the
> calls from a regular page call from the browser vs an ajax call.  They
> both make a request that goes to a servlet or jsp and they both get
> response data back from tomcat.  But the session times out if I don't do
> a real page call.  What am I missing?
>

Nothing!
Any request that uses the HttpSession
(HttpServletRequest.getHttpSession(true)) will "touch" the session and
update its "lastUsed" time, so it should not time out during active usage
of the application.
There must be something else that invalidates the http session.


>
> I really don't want to change the 'inactivity' timeout.  I just need to
> figure out how to tell tomcat with all of the ajax calls that the
> session is still very active.  Is there an easy solution?
>
> Thx
>
> Jerry
>
> BTW... TC 8.5
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Issue in CSS render with tomcat version 8.5.58 onwards

[Martin Grigorov]

Hi,

On Sat, Mar 13, 2021, 08:16 Shiv Garg  wrote:

> Hi Team,
> we have clients reporting issues with tomcat 8.5.58 version onwards,
> stating that CSS is not loading properly in chrome/FF browser.
>

Please explain more what exactly does not work.
Even better create a simple application that demonstrates the issue.


> Can you please help me check if there is any change in this release
> onwards which could impact it.
>
> in change logs i can see below change , kindly verify if this can make any
> impact.
>
>- [image: Fix:] Extracted CSS styles from the Manager we application
>for better code maintenance and replaced the GIF logo with SVG. (isapir)
>
>
No, this is not the problem.
This changes modified the Manager application that you may use to
deploy/start/stop your applications.

Regards,
Martin


> Regards
> Shiv Garg
>

Re: TOMCAT 10

[Martin Nybo Andersen]

On fredag den 12. februar 2021 11.41.26 CET KARTHIK SHIVAKUMAR wrote:
> Spec :
> Oracle Jdk15.1
> Tomcat  : 10.0.2
> Eclipse IDE :20201210-1552
> 
> Question   :   *A simple JSP with implicit Servlet API  Object as follows
> is not Working*
> 
> ServletContext sessnCntxt = getServletConfig().getServletContext();
> 
> Tested by deploying on the container, gives error
> 
> Can some body tell me What changes need to be done to
> 
> 
> Thx in advance
> 
> *N.S.KARTHIK*

Hi,

Which error does it give?
Do you have a stack trace?

Regards,
Martin


-- 
Time flies like the wind. Fruit flies like a banana.




-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Is ARM64 architecture officially supported ?

[Martin Grigorov]

Hi Chris,

On Thu, Feb 11, 2021 at 12:13 AM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> Martin,
>
> On 4/20/20 03:28, Martin Grigorov wrote:
> > Hi Michael,
> >
> > On Sun, Apr 19, 2020 at 9:08 PM Michael Osipov 
> wrote:
> >
> >> Am 2020-04-19 um 19:51 schrieb Martin Grigorov:
> >>> Hi Emilio,
> >>>
> >>> On Fri, Apr 17, 2020 at 2:14 PM Emilio Fernandes <
> >>> emilio.fernande...@gmail.com> wrote:
> >>>
> >>>> Hola Tomcat community!
> >>>>
> >>>> We consider using AWS Graviton [1] based instances which use ARM64
> >>>> processors for our backend services.
> >>>> I've googled around and found [2] saying that Tomcat is being tested
> on
> >>>> ARM64 architecture at TravisCI! This is great!
> >>>> Does this mean that Tomcat is officially supported on ARM64 ? I was
> not
> >>>> able to find any specific documentation listing which platforms are
> >>>> officially supported.
> >>>>
> >>>> Does anyone from the community have any experience with Tomcat/HTTPD
> on
> >>>> ARM64 in production ?
> >>>>
> >>>
> >>> I work a lot with ARM64 servers lately!
> >>> So far we didn't face any issue related to ARM64 & Tomcat in our
> >>> application!
> >>> I've introduced the testing on TravisCI that you have found. In
> addition
> >>> for my daily job we have created a nightly test setup:
> >>> - build and test Tomcat 9.x (as at Travis)
> >>> - run some smoke tests with our application
> >>> - run some performance tests on x86_64 and ARM64 VMs with similar
> >> hardware
> >>> specs. The results for both architectures are very similar. One thing
> >> that
> >>> still bothers me is that the throughput on HTTPS is 3 times less than
> >> HTTP.
> >>
> >> OpenSSL or SunJSSE?
> >>
> >
> > Both.
> > With OpenSSL it is slightly faster than with JSSE, but again around 3
> times
> > slower than HTTP.
> >
> > Here are some details:
> > Tomcat: 9.x nightly build
> > tcnative: nightly build from master branch
> > JDK: AdoptJDK 14
> > OpenSSL 1.1.1  11 Sep 2018 (this comes from Ubuntu 18.04 repos. I will
> try
> > with a newer build)
> > JMeter: 5.2.1
> > The application is Spring Boot 2.2.6 with REST GET/PUT/POST/DELETE
> > endpoints that simply sets/gets data to/from Memcached. No usage of
> Spring
> > Security! No redirects from HTTP to HTTPS!
> >
> > Let me know if you need more information!
>
> Coming back to this almost a year later. :)
>

Thanks for doing it! I've forgot to update this thread with the problem and
the solution!


>
> My guess about the TLS performance is that your x86 chips have built-in
> hardware support for crypto primitives used by the cipher suite(s) you
> are testing, and your ARM64 chips do not.
>
> Either that, or neither OpenSSL nor Java are able to use those ARM64
> hardware features because they are not aware of them (yet, I would guess).
>
> You would have to get the specs of the chips to find out whether
> hardware acceleration is even a possibility, and then approach the
> crypto providers (e.g. Oracle, OpenSSL) with questions about how to
> enable those hardware-supported routines.
>

At the end the problem was in Apache JMeter 5.2.1 - the tool I used for the
load testing back then.
As I expained here
<https://martin-grigorov.medium.com/compare-apache-tomcat-performance-on-x86-64-and-arm64-cpu-architectures-aacfbb0b5bb6>
one needs to add  --jmeterproperty
httpclient4.validate_after_inactivity=4900 and --jmeterproperty
httpclient4.time_to_live=12 to JMeter properties, otherwise it was
closing the connection and due to the TLS handshakes it was giving terrible
results. Those settings are default in JMeter 5.3.0+.

Apart from that there is also https://github.com/kunpengcompute/KAE for
HiSilicon/Kunpeng chips, to get even more from OpenSSL on this hardware.

Martin


>
> -chris
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat on Amazon's Graviton2

[Martin Grigorov]

Hi Chen,

On Wed, Feb 10, 2021 at 6:00 PM Chen Levy  wrote:

> Hello Everyone
>
> We've been running our Staging environment on ARM64 based servers (Amazon
> EC2 T4g) for a while now, and plan to use similar machines to replace our
> Production environment's x86-64 based servers in the near future.
> All of our servers are based on Amazon Linux 2 with AdoptOpenJDK 15.0.2
> and Tomcat 9.
> Measured with our application, performance between x86-64 and ARM64 with
> two similarly sized machines (e.g. T3.large vs T4g.large) is up by an
> average of 22% on the T4g's.
> These numbers are largely academic, and will stay that way until we
> actually deploy a few machines in Production.
> The ARM64 machines are also cheaper by about 20%, so if all goes well,
> there aren't real downsides to the move.
>
> I read this correspondence about a year ago:
> https://marc.info/?l=tomcat-user=2=1=arm64=b
>
> I'd like to ask the community if anyone has production experience with
> Tomcat and ARM64/Graviton2, and whether there are specific issues I should
> beware of.
>

I have experience on ARM64 architecture but on Huawei HiSilicon chips
(Kunpeng), not Graviton2.
TLDR - everything runs just fine! :-)

In the beginning we had some bad performance results but they were caused
by the load testing tools -
https://martin-grigorov.medium.com/compare-apache-tomcat-performance-on-x86-64-and-arm64-cpu-architectures-aacfbb0b5bb6
Initially I used Apache JMeter <https://jmeter.apache.org/> 5.2.1 and it
had the issue of closing the connection after each request. This caused
very bad results for HTTPS. JMeter team helped me by suggesting two
settings which fixed the problem. Those settings are now enabled by default
in 5.3.0+
Then I replaced JMeter with wrk <https://github.com/wg/wrk> - it gives much
better throughput results than JMeter! But at some point I realized (with
the help of Poul Henning Kamp from Varnish Cache team) that it has a nasty
bug in calculating the statistics (
https://martin-grigorov.medium.com/compare-varnish-cache-performance-on-x86-64-and-aarch64-cpu-architectures-cef5ad5fee5f
)
So I moved to Vegeta <https://github.com/tsenart/vegeta> ! It is almost
perfect for my needs. It has just one issue
<https://github.com/tsenart/vegeta/issues/540> - Golang http library sends
RST_STREAM frame after the END_STREAM for HTTP/2. But Mark Thomas improved
Tomcat to keep lightweight versions of the HTTP2 streams for some more time
and this worked around this issue.

For my tests I used two *similar* VMs - one x86_64 and one aarch64. The
have almost identical hardware but due to the different architectures not
exactly the same.
https://martin-grigorov.medium.com/comparing-the-performance-of-several-http2-web-servers-fb5f3787532a
is
my latest blog article related to Tomcat and there x86_64 still won. In the
same tests Netty performed better on aarch64 (same hardware, JDK and
OpenSSL versions!). I guess the reason is in the custom EPoll impl that
Netty uses, but I am not sure.

Apart from Tomcat most of the problems related to moving to ARM64 we found
were related to file format (Protobuf, Parquet) and file compression
(Brotli) libraries because they use native libraries.
The only big problem with pure Java (well, Scala in this case) I am aware
of was in Scala Cats library:
- https://github.com/typelevel/cats-effect/releases/tag/v3.0.0-M4
- code with explanation from OpenJDK expert:
https://gist.github.com/RaasAhsan/8e3554a41e07068536425ca0de46c9e8

Good luck!

Martin


>
> Thanks
> Chen
>
>
>

Re: How to completely deactivate JSESSIONID cookie entry (in Chrome's Cookie list)

[Martin Grigorov]

On Fri, Jan 29, 2021 at 12:03 PM Tillmann Schulz
 wrote:

> Hi there,
>
> I am using Tomcat 8.5.58 and have a problem with JSESIONID cookie.
> It should be possible to completly deactivate the jsessionid cookie with
> the following code:
>
> <%@ page session="false" %>
>

This says that this particular JSP does not create an HTTP session but if
any other JSP of your app creates one then it will be transfered for all
requests until it is either deleted or it expires.


>
> If you do that and call the JSP, there is in no entry under cookies in
> google chrome's cookie list (if you press F12 in chrome) .
> But in chrome's cookie list there is a host entry for the hostname of the
> JSP.
>

What is the name of this cookie ?


> This is confusing, because there are no cookies set.
>
> I am not sure if this is a tomcat issue, but under firefox it also looks
> simular.
>
> Is it possible to completly deactivate cookies? There shouldn't be any
> hostname visible in the cookie view.
>
> Thank you for your help in advance
>
> Tillmann
>
>
>
>

Re: Jars in Tomcat lib directory

[Martin Grigorov]

Hi,

On Mon, Jan 18, 2021 at 10:38 PM Jonnalagadda, Swathi (External) <
swathi.jonnalaga...@xerox.com> wrote:

> The application was an age old application and was built using ant which
> means all required jars will be packaged.
>
> As part of testing the application after upgrading to Java 1.8 and
> deployed in tomcat 9, I realized that couple of additional jars are
> required during runtime.
>
> I then added these jars to Tomcat lib directory and observed that it is
> not picking up these unless I add it to classpath
>

Classes in jar files in $CATALINA_BASE/lib are available for all web
applications!
How do you "add to the classpath" ? I have the feeling you do something in
your bin/setenv.sh that breaks the default configuration.


>
> Regards
> Swathi
>
> -Original Message-
> From: Mounika Reddy [mailto:spidermai...@gmail.com]
> Sent: Monday, January 18, 2021 11:49 PM
> To: Tomcat Users List
> Subject: Re: Jars in Tomcat lib directory
>
> CAUTION:   This email originated from outside the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
> It means your tomcat setup is screwed up. You can actually package in your
> application as a dependency and see if it picks up. If not, then post the
> logs here to understand what's going on
>
> On Mon, Jan 18, 2021, 1:09 PM Jonnalagadda, Swathi (External) <
> swathi.jonnalaga...@xerox.com> wrote:
>
> > I am not sure if I am missing some configuration setting but what I
> > see is unless I add externally added jars under tomcat/lib to
> > classpath I don’t see them considerd by Tomcat irrespective of wether
> they are needed or not.
> >
> > For example I added ojdbc14.jar to Tomcat lib directory assuming that
> > it will consider it in classpath but until I specified the path of the
> > jar in classpath, the server wasn’t considering the jar.
> >
> > Please advise.
> >
> > -Original Message-
> > From: Mounika Reddy [mailto:spidermai...@gmail.com]
> > Sent: Monday, January 18, 2021 6:32 PM
> > To: Tomcat Users List
> > Subject: Re: Jars in Tomcat lib directory
> >
> > CAUTION:   This email originated from outside the organization. Do not
> > click links or open attachments unless you recognize the sender and
> > know the content is safe.
> >
> > It will provided if your application requires.
> >
> > On Mon, Jan 18, 2021, 7:54 AM Jonnalagadda, Swathi (External) <
> > swathi.jonnalaga...@xerox.com> wrote:
> >
> > > Hi
> > >
> > > Doesn’t tomcat consider jar files under lib directory in classpath.
> > > Do we have to do any additional configuration for that?
> > >
> > >
> > > Thanks
> > > Swathi
> > >
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: Not able to make JSESSIONID cookie secure

[Martin Grigorov]

Hi Amit,

On Wed, Jan 6, 2021 at 11:15 AM Amit Khosla 
wrote:

> Hi,
>
> Thanks for the reply.
>
> We tried the settings on multiple machines. And found that the same
> configuration machines gave different results.
> We are getting multiple jsessionid cookies being created. In our
> application, we have a multi tenant application.
> For each tenant we have an nginx running calling the application url.
> URL being hit on browser is like
> *.myapp.com *
> We are able to see 2 JSESSIONID cookies being generated for this call on
> some machines.
> 1. domain: /myapp  which is having secure flag.
> 2. domain: /myapp/ which is NOT having secure flag.
>
> Strangely, in most machines, we are finding the second cookie being
> generated but the first cookie not generated.
> Only when we saw on one of the machines, the first cookie, we found that
> the secure jsessionid cookie being generated on that particular
> environment.
>
> The environment having 2 cookies and the one with one cookie are identical.
> Can you please help me fix the creation of the second cookie and also how
> to ensure that the first cookie is generated in all environments?
>

I think you issue is in Nginx config.
Please test first only with Tomcat and see whether it behaves propertly.
If it does then ask for more help at Nginx forums.
If it doesn't then please explain in more details how exactly you test it,
with configs, urls, etc.


>
>
> On Tue, Jan 5, 2021 at 1:24 AM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> > Mark,
> >
> > On 1/4/21 03:17, Mark Thomas wrote:
> > > On 04/01/2021 06:02, Amit Khosla wrote:
> > >> Hi,
> > >>
> > >> We are still facing this issue. Can someone please help us?
> > >
> > > In a clean 8.5.x install, session cookies are only marked as secure if
> > > the request that triggered the session creation is made over a secure
> > > channel (typically HTTPS).
> > >
> > > If you amend the session configuration in $CATALINA_BASE/conf/web.xml
> > from:
> > >
> > > 
> > >  30
> > > 
> > >
> > > to
> > >
> > > 
> > >  30
> > >  
> > >  true
> > >  
> > > 
> > >
> > > then session cookies will be generated with the secure flag whether or
> > > not the request that triggered the the session creation was made over a
> > > secure channel.
> > >
> > > Reviewing the thread:
> > >
> > > Are you sure you are amending the correct web.xml file? One way to
> check
> > > this is to make a deliberate error in the file and confirm that this
> > > error is reported when Tomcat starts.
> > >
> > > Note that you can only use  once in a web.xml file. If
> > > the web.xml file already contains a  element you must
> > > add to that existing element.
> > >
> > > Configuration in the application's web.xml file will override the
> global
> > > web.xml file. Make sure that the application's web.xml either does not
> > > specify a value for secure or specifies true.
> > >
> > > If you still have issues:
> > > - start with a clean Tomcat 8.5.x install
> > > - confirm that
> > >http://localhost:8080/examples/servlets/servlet/SessionExample
> > >generates a set-cookie header without the secure attribute
> > > - stop Tomcat
> > > - close the browser
> > > - amend conf/web.xml as above
> > > - start Tomcat
> > > - confirm that
> > >http://localhost:8080/examples/servlets/servlet/SessionExample
> > >generates a set-cookie header with the secure attribute
> > > - retest with your application
> > >
> > > You must close the browser between each request you expect to generate
> a
> > > session cookie to prevent any existing session from being used.
> >
> > You may be able to avoid this with either of:
> >
> > 1. "Private" browsing mode: use a new "private" tab/window each time
> >
> > 2. Under developer tools / Storage (you may have to poke around for
> > this), you should be able to see the cookies for the host and, if
> > appropriate, delete them.
> >
> > > If this test fails then you'll need to check the application source
> > > code. It is possible that the application is overriding your attempts
> to
> > > make the session cookie secure.
> >
> > +1
> >
> > Other possibilities include a reverse proxy where the client is using
> > HTTPS to communicate with the proxy, but HTTPS is not being used between
> > the proxy and the Tomcat server.
> >
> > -chris
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
>
> --
> Thanks & Regards
> Amit Khosla
>

Re: Resource interpreted as Stylesheet but transferred with MIME type text/html: "".

[Martin Grigorov]

On Mon, Nov 23, 2020 at 7:26 AM Huynh Nghia  wrote:

> - Request Headers : Accep:text/css
>

"Accep" header name misses "t" at the end


> - But Reponse : Content-type:text/html
> [image: Screenshot_8.png]
>
> Vào Th 2, 23 thg 11, 2020 vào lúc 11:47 Huynh Nghia <
> hhnghiai...@gmail.com> đã viết:
>
>>
>>
>> Vào Th 2, 23 thg 11, 2020 vào lúc 11:35 Huynh Nghia <
>> hhnghiai...@gmail.com> đã viết:
>>
>>>
>>>
>>> -- Forwarded message -
>>> Từ: Huynh Nghia 
>>> Date: Th 2, 23 thg 11, 2020 vào lúc 11:21
>>> Subject: Resource interpreted as Stylesheet but transferred with MIME
>>> type text/html: "".
>>> To: 
>>>
>>>
>>> Hi Tomcat dev,
>>>
>>> - I got an error with my app running on tomcat v9 64bit last modified
>>> (Win 10):
>>>   Example:
>>>
>>>  v9.0.27/ 
>>> 2019-10-11 08:46-   [image: [DIR]] v9.0.29/ 
>>> 
>>> 2019-11-21 09:11-   [image: [DIR]] v9.0.30/ 
>>> 
>>> 2019-12-12 08:22-   [image: [DIR]] v9.0.31/ 
>>> 
>>> 2020-03-10 23:16-   [image: [DIR]] v9.0.33/ 
>>> 
>>> 2020-03-16 09:37-   [image: [DIR]] v9.0.34/ 
>>> 
>>> 2020-04-08 15:43-   [image: [DIR]] v9.0.35/ 
>>> 
>>> 2020-05-11 11:19-   [image: [DIR]] v9.0.36/ 
>>> 
>>> 2020-06-07 19:01-   [image: [DIR]] v9.0.37/ 
>>> 
>>> 2020-07-05 18:31-   [image: [DIR]] v9.0.38/ 
>>> 
>>> 2020-11-05 18:47-
>>>
>>> [image: [DIR]] v9.0.39/
>>> 
>>>   2020-11-05 18:47 -
>>>
>>> v9.0.40/  
>>>2020-11-17 14:23
>>>
>>>
>>> [image: Screenshot_4.png]
>>> [image: Screenshot_5.png]
>>> [image: Screenshot_6.png]
>>>
>>>
>>> - But my app work with tomcat v9 last modified :
>>>   Example:
>>>
>>> v9.0.4/    
>>>   2018-01-22 14:51
>>>
>>> v9.0.5/    
>>>   2018-02-11 20:11
>>>
>>>
>>> - Technology : Angularjs 1.8, Jquery 3.5.1,Java Sevlet version 1.7 : 51.0
>>>
>>> - Please help me workaround this error or fix it on version tomcat
>>> v9.0.40/ 
>>>
>>> Thanks you so much.
>>>
>>> Best Regards,
>>>
>>> Nghia Huynh
>>>
>>>
>>>
>>>
>>>
>>>
>>>

Re: NGINX + tomcat 8.0.35 (110: Connection timed out)

[Martin Grigorov]

On Thu, Nov 12, 2020 at 2:40 PM Ayub Khan  wrote:

> Martin,
>
> Could you provide me a command which you want me to run and provide you the
> results which might help you to debug this issue ?
>

1) start your app and click around to load the usual FDs
2) lsof -p `cat /var/run/tomcat8.pid` > after_start.txt
3) load your app
4) lsof -p `cat /var/run/tomcat8.pid` > after_load.txt

you can analyze the differences in the files yourself before sending them
to us :-)


>
>
> On Thu, Nov 12, 2020 at 1:36 PM Martin Grigorov 
> wrote:
>
> > On Thu, Nov 12, 2020 at 10:37 AM Ayub Khan  wrote:
> >
> > > Martin,
> > >
> > > These are file descriptors, some are related to the jar files which are
> > > included in the web application and some are related to the sockets
> from
> > > nginx to tomcat and some are related to database connections. I use the
> > > below command to count the open file descriptors
> > >
> >
> > which type of connections increase ?
> > the sockets ? the DB ones ?
> >
> >
> > >
> > > watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
> > >
> >
> > you can also use lsof command
> >
> >
> > >
> > >
> > >
> > > On Thu, Nov 12, 2020 at 10:56 AM Martin Grigorov  >
> > > wrote:
> > >
> > > > On Wed, Nov 11, 2020 at 11:17 PM Ayub Khan 
> wrote:
> > > >
> > > > > Chris,
> > > > >
> > > > > I was load testing using the ec2 load balancer dns. I have
> increased
> > > the
> > > > > connector timeout to 6000 and also gave 32gig to the JVM of
> tomcat. I
> > > am
> > > > > not seeing connection timeout in nginx logs now. No errors in
> > > kernel.log
> > > > I
> > > > > am not seeing any errors in tomcat catalina.out.
> > > > > During regular operations when the request count is between 4 to 6k
> > > > > requests per minute the open files count for the tomcat process is
> > > > between
> > > > > 200 to 350. Responses from tomcat are within 5 seconds.
> > > > > If the requests count goes beyond 6.5 k open files slowly move up
> to
> > > > 2300
> > > > > to 3000 and the request responses from tomcat become slow.
> > > > >
> > > > > I am not concerned about high open files as I do not see any errors
> > > > related
> > > > > to open files. Only side effect of  open files going above 700 is
> the
> > > > > response from tomcat is slow. I checked if this is caused from
> > elastic
> > > > > search, aws cloud watch shows elastic search response is within 5
> > > > > milliseconds.
> > > > >
> > > > > what might be the reason that when the open files goes beyond 600,
> it
> > > > slows
> > > > > down the response time for tomcat. I tried with tomcat 9 and it's
> the
> > > > same
> > > > > behavior
> > > > >
> > > >
> > > > Do you know what kind of files are being opened ?
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Nov 3, 2020 at 9:40 PM Christopher Schultz <
> > > > > ch...@christopherschultz.net> wrote:
> > > > >
> > > > > > Ayub,
> > > > > >
> > > > > > On 11/3/20 10:56, Ayub Khan wrote:
> > > > > > > *I'm curious about why you are using all of cloudflare and ALB
> > and
> > > > > > > nginx.Seems like any one of those could provide what you are
> > > getting
> > > > > from
> > > > > > > all3 of them. *
> > > > > > >
> > > > > > > Cloudflare is doing just the DNS and nginx is doing ssl
> > termination
> > > > > >
> > > > > > What do you mean "Cloudflare is doing just the DNS?"
> > > > > >
> > > > > > So what is ALB doing, then?
> > > > > >
> > > > > > > *What is the maximum number of simultaneous requests that one
> > > > > > nginxinstance
> > > > > > > will accept? What is the maximum number of simultaneous
> > > > proxiedrequests
> > > > > > one
> > >

Re: NGINX + tomcat 8.0.35 (110: Connection timed out)

[Martin Grigorov]

On Thu, Nov 12, 2020 at 10:37 AM Ayub Khan  wrote:

> Martin,
>
> These are file descriptors, some are related to the jar files which are
> included in the web application and some are related to the sockets from
> nginx to tomcat and some are related to database connections. I use the
> below command to count the open file descriptors
>

which type of connections increase ?
the sockets ? the DB ones ?


>
> watch "sudo ls /proc/`cat /var/run/tomcat8.pid`/fd/ | wc -l"
>

you can also use lsof command


>
>
>
> On Thu, Nov 12, 2020 at 10:56 AM Martin Grigorov 
> wrote:
>
> > On Wed, Nov 11, 2020 at 11:17 PM Ayub Khan  wrote:
> >
> > > Chris,
> > >
> > > I was load testing using the ec2 load balancer dns. I have increased
> the
> > > connector timeout to 6000 and also gave 32gig to the JVM of tomcat. I
> am
> > > not seeing connection timeout in nginx logs now. No errors in
> kernel.log
> > I
> > > am not seeing any errors in tomcat catalina.out.
> > > During regular operations when the request count is between 4 to 6k
> > > requests per minute the open files count for the tomcat process is
> > between
> > > 200 to 350. Responses from tomcat are within 5 seconds.
> > > If the requests count goes beyond 6.5 k open files slowly move up  to
> > 2300
> > > to 3000 and the request responses from tomcat become slow.
> > >
> > > I am not concerned about high open files as I do not see any errors
> > related
> > > to open files. Only side effect of  open files going above 700 is the
> > > response from tomcat is slow. I checked if this is caused from elastic
> > > search, aws cloud watch shows elastic search response is within 5
> > > milliseconds.
> > >
> > > what might be the reason that when the open files goes beyond 600, it
> > slows
> > > down the response time for tomcat. I tried with tomcat 9 and it's the
> > same
> > > behavior
> > >
> >
> > Do you know what kind of files are being opened ?
> >
> >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Nov 3, 2020 at 9:40 PM Christopher Schultz <
> > > ch...@christopherschultz.net> wrote:
> > >
> > > > Ayub,
> > > >
> > > > On 11/3/20 10:56, Ayub Khan wrote:
> > > > > *I'm curious about why you are using all of cloudflare and ALB and
> > > > > nginx.Seems like any one of those could provide what you are
> getting
> > > from
> > > > > all3 of them. *
> > > > >
> > > > > Cloudflare is doing just the DNS and nginx is doing ssl termination
> > > >
> > > > What do you mean "Cloudflare is doing just the DNS?"
> > > >
> > > > So what is ALB doing, then?
> > > >
> > > > > *What is the maximum number of simultaneous requests that one
> > > > nginxinstance
> > > > > will accept? What is the maximum number of simultaneous
> > proxiedrequests
> > > > one
> > > > > nginx instance will make to a back-end Tomcat node? Howmany nginx
> > nodes
> > > > do
> > > > > you have? How many Tomcat nodes?  *
> > > > >
> > > > > We have 4 vms each having nginx and tomcat running on them and each
> > > > tomcat
> > > > > has nginx in front of them to proxy the requests. So it's one Nginx
> > > > > proxying to a dedicated tomcat on the same VM.
> > > >
> > > > Okay.
> > > >
> > > > > below is the tomcat connector configuration
> > > > >
> > > > >  > > > > connectionTimeout="6" maxThreads="2000"
> > > > >
>  protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > URIEncoding="UTF-8"
> > > > > redirectPort="8443" />
> > > >
> > > > 60 seconds is a *long* time for a connection timeout.
> > > >
> > > > Do you actually need 2000 threads? That's a lot, though not insane.
> > 2000
> > > > threads means you expect to handle 2000 concurrent (non-async,
> > > > non-Wewbsocket) requests. Do you need that (per node)? Are you
> > expecting
> > > > 8000 concurrent requests? Does your load-balancer understand the
> > > > topography and current-load on any given

Re: NGINX + tomcat 8.0.35 (110: Connection timed out)

[Martin Grigorov]

On Wed, Nov 11, 2020 at 11:17 PM Ayub Khan  wrote:

> Chris,
>
> I was load testing using the ec2 load balancer dns. I have increased the
> connector timeout to 6000 and also gave 32gig to the JVM of tomcat. I am
> not seeing connection timeout in nginx logs now. No errors in kernel.log I
> am not seeing any errors in tomcat catalina.out.
> During regular operations when the request count is between 4 to 6k
> requests per minute the open files count for the tomcat process is between
> 200 to 350. Responses from tomcat are within 5 seconds.
> If the requests count goes beyond 6.5 k open files slowly move up  to 2300
> to 3000 and the request responses from tomcat become slow.
>
> I am not concerned about high open files as I do not see any errors related
> to open files. Only side effect of  open files going above 700 is the
> response from tomcat is slow. I checked if this is caused from elastic
> search, aws cloud watch shows elastic search response is within 5
> milliseconds.
>
> what might be the reason that when the open files goes beyond 600, it slows
> down the response time for tomcat. I tried with tomcat 9 and it's the same
> behavior
>

Do you know what kind of files are being opened ?


>
>
>
>
>
>
> On Tue, Nov 3, 2020 at 9:40 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> > Ayub,
> >
> > On 11/3/20 10:56, Ayub Khan wrote:
> > > *I'm curious about why you are using all of cloudflare and ALB and
> > > nginx.Seems like any one of those could provide what you are getting
> from
> > > all3 of them. *
> > >
> > > Cloudflare is doing just the DNS and nginx is doing ssl termination
> >
> > What do you mean "Cloudflare is doing just the DNS?"
> >
> > So what is ALB doing, then?
> >
> > > *What is the maximum number of simultaneous requests that one
> > nginxinstance
> > > will accept? What is the maximum number of simultaneous proxiedrequests
> > one
> > > nginx instance will make to a back-end Tomcat node? Howmany nginx nodes
> > do
> > > you have? How many Tomcat nodes?  *
> > >
> > > We have 4 vms each having nginx and tomcat running on them and each
> > tomcat
> > > has nginx in front of them to proxy the requests. So it's one Nginx
> > > proxying to a dedicated tomcat on the same VM.
> >
> > Okay.
> >
> > > below is the tomcat connector configuration
> > >
> > >  > > connectionTimeout="6" maxThreads="2000"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > URIEncoding="UTF-8"
> > > redirectPort="8443" />
> >
> > 60 seconds is a *long* time for a connection timeout.
> >
> > Do you actually need 2000 threads? That's a lot, though not insane. 2000
> > threads means you expect to handle 2000 concurrent (non-async,
> > non-Wewbsocket) requests. Do you need that (per node)? Are you expecting
> > 8000 concurrent requests? Does your load-balancer understand the
> > topography and current-load on any given node?
> >
> > > When I am doing a load test of 2000 concurrent users I see the open
> files
> > > increase to 10,320 and when I take thread dump I see the threads are
> in a
> > > waiting state.Slowly as the requests are completed I see the open files
> > > come down to normal levels.
> >
> > Are you performing your load-test against the CF/ALB/nginx/Tomcat stack,
> > or just hitting Tomcat (or nginx) directly?
> >
> > Are you using HTTP keepalive in your load-test (from the client to
> > whichever server is being contacted)?
> >
> > > The output of the below command is
> > > sudo cat /proc/sys/kernel/pid_max
> > > 131072
> > >
> > > I am testing this on a c4.8xlarge VM in AWS.
> > >
> > > below is the config I changed in nginx.conf file
> > >
> > > events {
> > >  worker_connections 5;
> > >  # multi_accept on;
> > > }
> >
> > This will allow 50k incoming connections, and Tomcat will accept an
> > unbounded number of connections (for NIO connector). So limiting your
> > threads to 2000 only means that the work of each request will be done in
> > groups of 2000.
> >
> > > worker_rlimit_nofile 3;
> >
> > I'm not sure how many connections are handled by a single nginx worker.
> > If you accept 50k connections and only allow 30k file handles, you may
> > have a problem if that's all being done by a single worker.
> >
> > > What would be the ideal config for tomcat and Nginx so this setup on
> > > c4.8xlarge vm could serve at least 5k or 10k requests simultaneously
> > > without causing the open files to spike to 10K.
> >
> > You will never be able to serve 10k simultaneous requests without having
> > 10k open files on the server. If you mean 10k requests across the whole
> > 4-node environment, then I'd expect 10k requests to open (roughly) 2500
> > open files on each server. And of course, you need all kinds of other
> > files open as well, from JAR files to DB connections or other network
> > connections.
> >
> > But each connection needs a file descriptor, full stop. If you need 

Re: Timeout waiting to read data from client

[Martin Grigorov]

On Thu, Nov 12, 2020 at 12:34 AM Jerry Malcolm 
wrote:

> On 11/9/2020 11:05 AM, Jerry Malcolm wrote:
> >
> > On 11/9/2020 3:10 AM, Mark Thomas wrote:
> >> On 08/11/2020 01:33, Jerry Malcolm wrote:
> >>> On 11/7/2020 6:56 PM, Christopher Schultz wrote:
>  Jerry,
> 
>  On 11/6/20 19:49, Jerry Malcolm wrote:
> > I have a relatively new environment with a standalone tomcat (8.5)
> > running on an AWS Linux2 EC2.  I'm not using HTTPD/AJP. Its a direct
> > connection to port 443.  (Well technically, I have firewallD in the
> > flow in order to route the protected port 80 to port 8080 and 443 to
> > 8443 for TC).
> >
> > I am doing some stress testing on the server and failing miserably.
> > I am sending around 130 ajax calls in rapid succession using HTTP/2.
> > These are all very simple small page (JSP) requests.  Not a lot of
> > processing required. The first ~75 requests process normally.  Then
> > everything hangs up.  In the tomcat logs I'm getting a bunch of
> > "Timeout waiting to read data from client" exceptions. And in the
> > stacktrace for these exceptions, they are all occurring when I'm
> > trying to access a parameter from the request.  Looking at the
> > request network timing in the browser console, I see a bunch of the
> > requests returning in typical time of a few milliseconds. Then
> > another large block of requests that all start returning around 4
> > seconds, then another block that wait until 8 seconds to return.
> > I've tried firefox and chrome with the same results.
> >
> > I've been using httpd in front of TC for years.  So this is the first
> > time I'm running TC standalone.  It is very likely I've got some
> > parameters set horribly wrong.  But I have no clue where to start.
> > This is not a tiny EC2, and my internet connection is not showing any
> > signs of problems.  So I really don't think this is a
> > performance-related problem.  The problem is very consistent and
> > reproducible with the same counts of success/failure calls. What
> > could be causing the "Timeout waiting to read data from client" after
> > 75 calls, and then cause blocks of calls to wait 4 seconds, 8
> > seconds, etc before responding?  I really need to handle more
> > simultaneous load that this is currently allowing.
> >
> > Thanks in advance for the education.
>  Are you using HTTP Keepalives on your connections? Are you actually
>  re-using those connections in your test? What is your keepalive
>  timeout on your . Actually, what is your whole 
>  configuration?
> 
>  -chris
> 
> >>> Hi Chris, here are my two connector definitions from server.xml:
> >>>
> >>>   >>>port="8080"
> >>>protocol="HTTP/1.1"
> >>>connectionTimeout="2"
> >>>redirectPort="443" />
> >>>
> >>>   >>>port="8443"
> >>>maxThreads="150"
> >>>connectionTimeout="2"
> >>>SSLEnabled="true"
> >>>scheme="https"
> >>>secure="true"
> >>>clientAuth="false"
> >>>SSLCertificateFile="ssl/a.com/cert.pem"
> >>>SSLCertificateChainFile="ssl/a.com/chain.pem"
> >>>SSLCertificateKeyFile="ssl/a.com/privkey.pem">
> >>>   >>> className="org.apache.coyote.http2.Http2Protocol" />
> >>>  
> >> How are you stress testing this? All on a single HTTP/2 connection or
> >> multiple connections? With which tool?
> >>
> >> You might want to test HTTP/1.1 requests (with and without TLS) to see
> >> if the problem is specific to HTTP/2 or TLS as that should help narrow
> >> down the root cause.
> >>
> >> Mark
> >
> > Hi Mark, technically it's not a 'designed' stress test.  It's real
> > production code that just happens to stress the server more than
> > usual.  It's just a page that makes a bunch of ajax calls, and the
> > responses to each of those issue a second ajax call.
> >
> > If you don't see anything obvious in my configuration, we will
> > definitely pursue the http/1.1 options, etc.  I just wanted to
> > eliminate the chance of obvious 'pilot error' before digging deeper.
> >
> > Specifically, where is that error detected in the TC flow? In my logs
> > it fails on getting request parameters.  It sounds like the input
> > reader for the request is getting blocked.But the first part of
> > the request is getting in since it does route to the appropriate JSP.
> > Just seems strange that the http/2 or ssl layers would let half of the
> > request in and then block the rest of the request.  The browser
> > appears to be sending everything.  And it fails the same using firefox
> > or chrome.  Any ideas?
> >
> > Thx
> >
> >
> Update on this.  One of our clients got ERR_HTTP2_SERVER_REFUSED_STREAM
> after things locked up.  I removed the http2 'upgrade protocol' line
> from my connector, and everything works.  So it's apparently something

Re: Tomcat - Sentry configuration problem

[Martin Grigorov]

On Wed, Nov 11, 2020 at 3:16 PM Tamás Barta  wrote:

> Hi,
>
> > Why do you use 1.7.29 ?
>
> Because our company has a Sentry server supports this version of clients.
>
> > Do you specify this class name in logging.properties ? Is
> this class in 1.7.29 ?
>
> Yes and yes. If I add sentry jars to classpath, sentry works well.
> Exceptions are sent to the server and visible in web interface.
>
> > You just need to add it as a normal Maven dependency.
>
> Maven dependency is needed only If I would like to use Sentry in code. But
> I don't.
>
> > If you need to really put it on the bootstrap.
>
> When I say bootstrap I mean Tomcat bootstrap:
> https://tomcat.apache.org/tomcat-8.0-doc/class-loader-howto.html
> Which is standard java classpath, I see.
>
> BTW if I add jars to java bootclasspath, I got the same error.
>
> Just a very interesting test: if I add an empty JAR (no class files in it)
> file to setenv.sh (no Sentry jars at all) that I got the same error.
> It seems that adding an empty jar to classpath case this classloading
> problem :(
>

I've just tested this on 10.0.0-M9

Using CATALINA_BASE:   /home/martin/devel/apache-tomcat-10.0.0-M9
Using CATALINA_HOME:   /home/martin/devel/apache-tomcat-10.0.0-M9
Using CATALINA_TMPDIR: /home/martin/devel/apache-tomcat-10.0.0-M9/temp
Using JRE_HOME:/home/martin/devel/jdk-15
Using CLASSPATH:
:/home/martin/devel/apache-tomcat-10.0.0-M9/empty.jar:/home/martin/devel/apache-tomcat-10.0.0-M9/bin/bootstrap.jar:/home/martin/devel/apache-tomcat-10.0.0-M9/bin/tomcat-juli.jar
Using CATALINA_OPTS:
NOTE: Picked up JDK_JAVA_OPTIONS:
 --add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

$ cat bin/setenv.sh
export CLASSPATH=$CLASSPATH:$CATALINA_BASE/empty.jar

$ ll empty.jar
-rw-r- 1 martin martin 22 ное 11 15:48 empty.jar

$ jar --list -f empty.jar


There is no problem with loading this empty jar and there are no class
loading issues either.



>
> Regards, Tamas
>
>
>
>
>
>
> On Wed, Nov 11, 2020 at 12:33 PM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > On Wed, Nov 11, 2020 at 11:20 AM Tamás Barta 
> wrote:
> >
> > > Hi,
> > >
> > > The ways you wrote is not usable, because they cause the following
> > problem:
> > >
> > > Handler error
> > > java.lang.ClassNotFoundException: io.sentry.jul.SentryHandler
> > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> > > at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> > > at
> > >
> > >
> >
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:582)
> > > at
> > >
> > >
> >
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:524)
> > > at
> > >
> > >
> >
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:322)
> > > at java.util.logging.LogManager$3.run(LogManager.java:399)
> > > at java.util.logging.LogManager$3.run(LogManager.java:396)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at
> > >
> > >
> >
> java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396)
> > > at java.util.logging.LogManager.access$800(LogManager.java:145)
> > > at java.util.logging.LogManager$2.run(LogManager.java:345)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at
> > >
> > >
> >
> java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338)
> > > at
> > java.util.logging.LogManager.getLogManager(LogManager.java:378)
> > > at java.util.logging.Logger.demandLogger(Logger.java:448)
> > > at java.util.logging.Logger.getLogger(Logger.java:502)
> > > at
> > > org.apache.juli.logging.DirectJDKLog.(DirectJDKLog.java:67)
> > > at
> > > org.apache.juli.logging.DirectJDKLog.getInstance(DirectJDKLog.java:187)
> > > at
> > > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:117)
> > > at
> > > org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:141)
> > > at
> org.apache.juli.logging.LogFactory.getLog(LogFactory.java:196)

Re: Tomcat - Sentry configuration problem

[Martin Grigorov]

Hi,

On Wed, Nov 11, 2020 at 11:20 AM Tamás Barta  wrote:

> Hi,
>
> The ways you wrote is not usable, because they cause the following problem:
>
> Handler error
> java.lang.ClassNotFoundException: io.sentry.jul.SentryHandler
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
>
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:582)
> at
>
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:524)
> at
>
> org.apache.juli.ClassLoaderLogManager.readConfiguration(ClassLoaderLogManager.java:322)
> at java.util.logging.LogManager$3.run(LogManager.java:399)
> at java.util.logging.LogManager$3.run(LogManager.java:396)
> at java.security.AccessController.doPrivileged(Native Method)
> at
>
> java.util.logging.LogManager.readPrimordialConfiguration(LogManager.java:396)
> at java.util.logging.LogManager.access$800(LogManager.java:145)
> at java.util.logging.LogManager$2.run(LogManager.java:345)
> at java.security.AccessController.doPrivileged(Native Method)
> at
>
> java.util.logging.LogManager.ensureLogManagerInitialized(LogManager.java:338)
> at java.util.logging.LogManager.getLogManager(LogManager.java:378)
> at java.util.logging.Logger.demandLogger(Logger.java:448)
> at java.util.logging.Logger.getLogger(Logger.java:502)
> at
> org.apache.juli.logging.DirectJDKLog.(DirectJDKLog.java:67)
> at
> org.apache.juli.logging.DirectJDKLog.getInstance(DirectJDKLog.java:187)
> at
> org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:117)
> at
> org.apache.juli.logging.LogFactory.getInstance(LogFactory.java:141)
> at org.apache.juli.logging.LogFactory.getLog(LogFactory.java:196)
> at
> org.apache.catalina.startup.Bootstrap.(Bootstrap.java:52)
> 10-Nov-2020 17:44:11.137 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server version:
>  Apache Tomcat/8.5.23
> 10-Nov-2020 17:44:11.139 INFO [main]
> org.apache.catalina.startup.VersionLoggerListener.log Server built:
>  Sep 28 2017 10:30:11 UTC
> ...
>
> The Sentry log handler must be in Boostrap jars, it must be loaded before
> web application and before jars in $CATALINA_BASE/lib. If I am right, the
> best way to do that is by setting it in setenv.sh.
>

I haven't used Sentry before so I just looked it up.
https://docs.sentry.io/platforms/java/usage/ says that the current version
is 3.1.1. Why do you use 1.7.29 ? There is no io.sentry.jul.SentryHandler
class in 3.1.1. Do you specify this class name in logging.properties ? Is
this class in 1.7.29 ?
There is nothing about "bootstrap" on
https://docs.sentry.io/platforms/java/usage/ or at
https://docs.sentry.io/platforms/java/guides/spring-boot/. You just need to
add it as a normal Maven dependency.
If you need to really put it on the bootstrap path then you need to use
-Xbootclasspath:...
By using CLASSPATH=$CLASSPATH:... you actually *append* jars to the
classpath, while you sound like you need to *prepend* to it.


>
> I can't imagine how could it be possible without setting sentry jar the web
> application works well, there is no problem, but if I add sentry-1.7.29.jar
> to classpath then ServletContainerInitializer won't be found by Catalina.
> This class is only in $CATALINA_BASE/lib/servlet-api.jar.
>
> Thanks, Tamás
>
> On Wed, Nov 11, 2020 at 9:46 AM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > On Wed, Nov 11, 2020 at 10:36 AM Tamás Barta 
> wrote:
> >
> > > Hi,
> > >
> > > Now it is a Spring Boot web application project without embedded
> tomcat.
> > > There is no library in war with javax.servlet api classes. There error
> is
> > > the same:
> > >
> > > 11-Nov-2020 09:34:06.719 SEVERE [localhost-startStop-1]
> > > org.apache.catalina.core.ContainerBase.addChildInternal
> > > ContainerBase.addChild: start:
> > >  org.apache.catalina.LifecycleException: Failed to start component
> > >
> > >
> >
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ReplicationService-0.0.1-SNAPSHOT]]
> > > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
> > > at
> > >
> > >
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
> > > at

Re: Tomcat - Sentry configuration problem

[Martin Grigorov]

Hi,

On Wed, Nov 11, 2020 at 10:36 AM Tamás Barta  wrote:

> Hi,
>
> Now it is a Spring Boot web application project without embedded tomcat.
> There is no library in war with javax.servlet api classes. There error is
> the same:
>
> 11-Nov-2020 09:34:06.719 SEVERE [localhost-startStop-1]
> org.apache.catalina.core.ContainerBase.addChildInternal
> ContainerBase.addChild: start:
>  org.apache.catalina.LifecycleException: Failed to start component
>
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/ReplicationService-0.0.1-SNAPSHOT]]
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
> at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988)
> at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NoClassDefFoundError:
> javax/servlet/ServletContainerInitializer
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
>
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1277)
> at
>
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1119)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
>
> org.apache.catalina.startup.WebappServiceLoader.loadServices(WebappServiceLoader.java:188)
> at
>
> org.apache.catalina.startup.WebappServiceLoader.load(WebappServiceLoader.java:159)
> at
>
> org.apache.catalina.startup.ContextConfig.processServletContainerInitializers(ContextConfig.java:1622)
> at
>
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1135)
> at
>
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:775)
> at
>
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
> at
>
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5105)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> ... 10 more
> Caused by: java.lang.ClassNotFoundException:
> javax.servlet.ServletContainerInitializer
>

Definitely you mess up the classpath and there are none or more than one
servlet-api.jar in the classpath.

What happens if you put your sentry related jars in the .war file
(/WEB-INF/lib/) ? I.e. add them as proper Maven dependencies
Or if you put them in $CATALINA_BASE/lib ?
Why did you go with editing setenv.sh in first place ?


> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 38 more
>
> On Tue, Nov 10, 2020 at 11:28 PM Christopher Schultz <
> ch...@christopherschultz.net> wrote:
>
> > Tamás,
> >
> > On 11/10/20 15:42, Tamás Barta wrote:
> > > This is the content of setenv.sh:
> > >
> > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/sentry-1.7.29.jar
> > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/slf4j-api-1.7.24.jar
> > > CLASSPATH=$CLASSPATH:$CATALINA_BASE/sentry/jackson-core-2.8.10.jar
> > >
> > > The web application without this sentry configuration works well for
> > ages.
> > > But if I add the above configuration, it won't.
> > >
> > > Yes, it is a maven project, now I checked and I see that there are
> Tomcat
> > > jar-s in it:
> > >
> > > tomcat-annotations-api-8.5.23.jar  tomcat-embed-core-8.5.23.jar
> > >   tomcat-embed-el-8.5.23.jar  

Re: Can Tomcat 9 be FIPS compliant without OpenSSL?

[Martin Grigorov]

Hi,

On Fri, Nov 6, 2020 at 8:57 AM Avik Ray  wrote:

> Dear team,
> Sending this query again after subscribing to the mailing list. Sent
> it originally 3 days back, but just saw an error response in the spam
> folder asking to subscribe first.
>
> We are using Tomcat 9.0.37 x64 on Windows Server 2016 OS and the NIO
> connector with JSSE, without an underlying OpenSSL.
>
> As per Tomcat 9 docs, the only mention of FIPS compliant operation I
> see is in the config of APR lifecycle listener, with the expectation
> of an underlying OpenSSL implementation that can be set to FIPS
> enabled mode. Ref:
> https://tomcat.apache.org/tomcat-9.0-doc/config/listeners.html
>
> Is it possible to be FIPS compliant with the usage of Tomcat, without
> the above setting? We were thinking of using BouncyCastle FIPS as the
> underlying Java crypto provider instead of OpenSSL for multiple
> reasons.
>
> Are there any other dependencies Tomcat has on the underlying stack,
> besides that provided by a Java crypto provider like BC-FIPS, having a
> bearing on FIPS compliance?
>
> Please advise, as this is urgent for a FIPS compliance decision.
>

Please check the README of this project -
https://github.com/amitlpande/tomcat-9-fips
Amit Pande recently shared it here at users@.

Regards,
Martin


>
> Thanks,
> Avik Ray
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: GraalVM native image for embedded Tomcat

[Martin Grigorov]

On Fri, Oct 16, 2020 at 3:43 PM Martin Grigorov 
wrote:

> Hi,
>
> I am trying to create a GraalVM native image for an application that uses
> tomcat-embed-core but it fails with the following error:
>
> [INFO] --- native-image-maven-plugin:20.2.0:native-image (default) @
> tomcat-embedded ---
> [INFO] ImageClasspath Entry:
> org.apache.tomcat.embed:tomcat-embed-core:jar:9.0.39:compile
> (file:///home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar)
> [INFO] ImageClasspath Entry:
> org.apache.tomcat:tomcat-annotations-api:jar:9.0.39:compile
> (file:///home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar)
> [INFO] ImageClasspath Entry:
> info.mgsolutions:tomcat-embedded:jar:1.0-SNAPSHOT
> (file:///home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar)
> [INFO] Executing:
> /home/martin/devel/graalvm-ce-java11-20.2.0/bin/native-image -cp
> /home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar:/home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar:/home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar
> -H:Class=info.mgsolutions.tomcat.TomcatEmbedded
> [info.mgsolutions.tomcat.tomcatembedded:568255]classlist:   2,047.64
> ms,  0.96 GB
> [info.mgsolutions.tomcat.tomcatembedded:568255](cap): 852.60
> ms,  0.96 GB
> [info.mgsolutions.tomcat.tomcatembedded:568255]setup:   2,323.40
> ms,  0.96 GB
> Fatal error:java.util.MissingResourceException: Can't find bundle for base
> name org.apache.tomcat.util.descriptor.tld.LocalStrings, locale en_US
> at
> java.base/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055)
> at
> java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689)
> at
> java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593)
> at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:1284)
> at
> com.oracle.svm.util.ModuleSupport.getResourceBundleFallback(ModuleSupport.java:81)
> at
> com.oracle.svm.util.ModuleSupport.getResourceBundle(ModuleSupport.java:62)
> at
> com.oracle.svm.core.jdk.LocalizationFeature.addBundleToCache(LocalizationFeature.java:266)
> at
> com.oracle.svm.hosted.ResourcesFeature$ResourcesRegistryImpl.addResourceBundles(ResourcesFeature.java:92)
> at
> com.oracle.svm.core.configure.ResourceConfigurationParser.parseEntry(ResourceConfigurationParser.java:90)
> at
> com.oracle.svm.core.configure.ResourceConfigurationParser.parseTopLevelObject(ResourceConfigurationParser.java:71)
> at
> com.oracle.svm.core.configure.ResourceConfigurationParser.parseAndRegister(ResourceConfigurationParser.java:47)
> at
> com.oracle.svm.hosted.config.ConfigurationParserUtils.doParseAndRegister(ConfigurationParserUtils.java:132)
> at
> com.oracle.svm.hosted.config.ConfigurationParserUtils.lambda$parseAndRegisterConfigurations$3(ConfigurationParserUtils.java:119)
> at
> java.base/java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:212)
> at
> com.oracle.svm.hosted.config.ConfigurationParserUtils$1.tryAdvance(ConfigurationParserUtils.java:108)
> at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:326)
> at
> java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
> at
> java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
> at
> java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
> at
> java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.base/java.util.stream.IntPipeline.reduce(IntPipeline.java:491)
> at java.base/java.util.stream.IntPipeline.sum(IntPipeline.java:449)
> at
> com.oracle.svm.hosted.config.ConfigurationParserUtils.parseAndRegisterConfigurations(ConfigurationParserUtils.java:125)
> at
> com.oracle.svm

GraalVM native image for embedded Tomcat

[Martin Grigorov]

Hi,

I am trying to create a GraalVM native image for an application that uses
tomcat-embed-core but it fails with the following error:

[INFO] --- native-image-maven-plugin:20.2.0:native-image (default) @
tomcat-embedded ---
[INFO] ImageClasspath Entry:
org.apache.tomcat.embed:tomcat-embed-core:jar:9.0.39:compile
(file:///home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar)
[INFO] ImageClasspath Entry:
org.apache.tomcat:tomcat-annotations-api:jar:9.0.39:compile
(file:///home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar)
[INFO] ImageClasspath Entry:
info.mgsolutions:tomcat-embedded:jar:1.0-SNAPSHOT
(file:///home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar)
[INFO] Executing:
/home/martin/devel/graalvm-ce-java11-20.2.0/bin/native-image -cp
/home/martin/.m2/repository/org/apache/tomcat/embed/tomcat-embed-core/9.0.39/tomcat-embed-core-9.0.39.jar:/home/martin/.m2/repository/org/apache/tomcat/tomcat-annotations-api/9.0.39/tomcat-annotations-api-9.0.39.jar:/home/martin/git/http2-server-perf-tests/java/tomcat/target/tomcat-embedded-1.0-SNAPSHOT.jar
-H:Class=info.mgsolutions.tomcat.TomcatEmbedded
[info.mgsolutions.tomcat.tomcatembedded:568255]classlist:   2,047.64
ms,  0.96 GB
[info.mgsolutions.tomcat.tomcatembedded:568255](cap): 852.60
ms,  0.96 GB
[info.mgsolutions.tomcat.tomcatembedded:568255]setup:   2,323.40
ms,  0.96 GB
Fatal error:java.util.MissingResourceException: Can't find bundle for base
name org.apache.tomcat.util.descriptor.tld.LocalStrings, locale en_US
at
java.base/java.util.ResourceBundle.throwMissingResourceException(ResourceBundle.java:2055)
at
java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1689)
at
java.base/java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1593)
at java.base/java.util.ResourceBundle.getBundle(ResourceBundle.java:1284)
at
com.oracle.svm.util.ModuleSupport.getResourceBundleFallback(ModuleSupport.java:81)
at
com.oracle.svm.util.ModuleSupport.getResourceBundle(ModuleSupport.java:62)
at
com.oracle.svm.core.jdk.LocalizationFeature.addBundleToCache(LocalizationFeature.java:266)
at
com.oracle.svm.hosted.ResourcesFeature$ResourcesRegistryImpl.addResourceBundles(ResourcesFeature.java:92)
at
com.oracle.svm.core.configure.ResourceConfigurationParser.parseEntry(ResourceConfigurationParser.java:90)
at
com.oracle.svm.core.configure.ResourceConfigurationParser.parseTopLevelObject(ResourceConfigurationParser.java:71)
at
com.oracle.svm.core.configure.ResourceConfigurationParser.parseAndRegister(ResourceConfigurationParser.java:47)
at
com.oracle.svm.hosted.config.ConfigurationParserUtils.doParseAndRegister(ConfigurationParserUtils.java:132)
at
com.oracle.svm.hosted.config.ConfigurationParserUtils.lambda$parseAndRegisterConfigurations$3(ConfigurationParserUtils.java:119)
at
java.base/java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:212)
at
com.oracle.svm.hosted.config.ConfigurationParserUtils$1.tryAdvance(ConfigurationParserUtils.java:108)
at java.base/java.util.Spliterator.forEachRemaining(Spliterator.java:326)
at
java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
at
java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:734)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.IntPipeline.reduce(IntPipeline.java:491)
at java.base/java.util.stream.IntPipeline.sum(IntPipeline.java:449)
at
com.oracle.svm.hosted.config.ConfigurationParserUtils.parseAndRegisterConfigurations(ConfigurationParserUtils.java:125)
at
com.oracle.svm.hosted.ResourcesFeature.beforeAnalysis(ResourcesFeature.java:105)
at
com.oracle.svm.hosted.NativeImageGenerator.lambda$runPointsToAnalysis$7(NativeImageGenerator.java:693)
at
com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:70)
at
com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:693)
at
com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:555)
at
com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:468

Re: Deploying war, Negative Date exception

[Martin Grigorov]

Hi Peter,

On Mon, Oct 12, 2020 at 2:20 PM Peter Henderson 
wrote:

> Hello fellow tomcat users.
>
> My environment.
> Tomcat: 9.0.39
> Java: openjdk 11.0.8 2020-07-14
> OS: Ubuntu 18.04.5 LTS
>
> Source code [0]
>
> When deploying this war [1], by copying it into the webapps directory,
> I get this exception. [2]
> java.lang.IllegalArgumentException: Negative time
>
>
> I only started seeing this exception when I upgraded my projects build tool
> version
> from
> sbt.version=1.3.10
> to
> sbt.version=1.4.0
>
>
> Is this a tomcat bug, a build tool bug or most likely something I'm doing
> wrong?
>
> Thanks
> Peter.
>
>
> [0]
> https://github.com/bollinger/NegativeDate
>
> [1]
> https://github.com/bollinger/NegativeDate/blob/master/Negative.war
>
> [2]
> 2020-10-12 11:41:35.932 SEVERE oacs.HostConfig Error deploying web
> application archive [/home/peter/apache-tomcat-9.0.39/webapps/Negative.war]
> java.lang.IllegalStateException: Error starting child
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:720)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:706)
> at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:978)
> at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1848)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
> org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
> at
>
> java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118)
> at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:773)
> at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:427)
> at org.apache.catalina.startup.HostConfig.check(HostConfig.java:1620)
> at
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:305)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
>
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1151)
> at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
> at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
> at
>
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
> at
>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
>
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.catalina.LifecycleException: Failed to start
> component
>
> [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/Negative]]
> at
>
> org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
> at
>
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717)
> ... 24 more
> Caused by: java.lang.IllegalArgumentException: Negative time
> at java.base/java.io.File.setLastModified(File.java:1441)
>

The negative time comes from jarEntry.getTime() -
https://github.com/apache/tomcat/blob/d6e5d838a9565c0c4bfeca4396e6266d21a80288/java/org/apache/catalina/startup/ExpandWar.java#L167
I guess SBT creates the JAR/WAR so most probably SBT 1.4.0 is to blame here.

Martin


> at org.apache.catalina.startup.ExpandWar.expand(ExpandWar.java:169)
> at
>
> org.apache.catalina.startup.ContextConfig.fixDocBase(ContextConfig.java:820)
> at
>
> org.apache.catalina.startup.ContextConfig.beforeStart(ContextConfig.java:958)
> at
>
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:305)
> at
>
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
> at
>
> org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:182)
> ... 25 more
>
> --
> Peter Henderson
>

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

Hi Arshiya,


On Fri, Oct 9, 2020 at 2:33 PM Arshiya Shariff
 wrote:

> Hi,
>
> Mark , with the test runs that I performed over clean 9.0.x branch I was
> not able to reproduce this. But with 9.0.38 and the jars built from 9.0.x
> with hash: c8ec2d4cde3a31b0e9df9a30e7915d77ba725545  , with 700 or 1000
> users (connections) and on sending 1000 Requests per second (or even
> lesser) , payload of 16K  from JMeter I can see that this Exception occurs
> within few minutes of starting the test . The maxThreads configured in
> tomcat is 200 .
>
> How often do you see these errors in your test run?
> Randomly, at times 2 or 3 such traces.
>
> Do you have the other end of that stack trace?
> It is only the two lines that is recursively printed till the end about
> ~500 times in one trace  :
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>
> I see the trace starting with :
> Exception in thread "http-nio-x.y.z-1090-exec-107"
> java.lang.StackOverflowError
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:446)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>
> (OR)
>
> Exception in thread "http-nio-x.y.z-1090-exec-87"
> java.lang.StackOverflowError
> at sun.nio.ch.IOVecWrapper.get(IOVecWrapper.java:96)
> at sun.nio.ch.IOUtil.read(IOUtil.java:240)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
> at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> .
> .
> .
> .
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
>
> Is there anything that was fixed around this in latest 9.0.x branch ?
>

9.0.39 is being voted now at dev@ mailing list:

It can be obtained from:
https://dist.apache.org/repos/dist/dev/tomcat/tomcat-9/v9.0.39/
The Maven staging repo is:
https://repository.apache.org/content/repositories/orgapachetomcat-1281/

Give it a try and vote!


> Thanks and Regards
> Arshiya Shariff
>
> -Original Message-
> From: Mark Thomas 
> Sent: Monday, October 5, 2020 9:52 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: memory filled up fast on increasing the connections to
> 1000/2000 (Embedded tomcat 9.0.38)
>
> On 05/10/2020 10:56, Arshiya Shariff wrote:
> > Hi All,
> >
> > Thank you so much Mark .
> > We tested the jars built from latest 9.0.x  with 2000 / 5000 users
> > (connections) from JMeter , We see a very good improvement with the
> > heap usage
>
> Good news. As is the fact that the other errors have been cleared up.
>
> > But I see this exception printed multiple times , I am not sure why this
> occurs :
> > Exception in thread "http-nio-x.y.z-1234-exec-213"
> java.lang.StackOverflowError
> > at sun.nio.ch.IOUtil.read(IOUtil.java:240)
> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
> > at org.apache.tomcat.util.net
> .NioChannel.read(NioChannel.java:174)
> > at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> > at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> > at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> > at org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> > at org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> > at org.apache.tomcat.util.net
> 

Re: Fwd: Re: At least one JAR was scanned for TLDs yet contained no TLDs.

[Martin Grigorov]

On Thu, Oct 8, 2020 at 3:31 PM Raivo Rebane  wrote:

> Hello
>
> if I start standalone tomcat program looks like:
>
> 17868 ?Sl 0:02 /usr/lib/jvm/default-java/bin/java
> -Djava.util.logging.config.file=/opt/tomcat/latest/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djava.security.egd=file:///dev/urandom -Djava.awt.headless=true
> -Djdk.tls.ephemeralDHKeySize=2048
> -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
> -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Xms512M
> -Xmx1024M -server -XX:+UseParallelGC -Dignore.endorsed.dirs= -classpath
> /opt/tomcat/latest/bin/bootstrap.jar:/opt/tomcat/latest/bin/tomcat-juli.jar
>
> -Dcatalina.base=/opt/tomcat/latest -Dcatalina.home=/opt/tomcat/latest
> -Djava.io.tmpdir=/opt/tomcat/latest/temp
> org.apache.catalina.startup.Bootstrap start
>
> No jars is added to classpath
>

What are you trying to do?
Why do you start it this way and what issue do you think you have with the
TLDs ?


>
> Raivo
>
> On 08.10.20 13:59, Raivo Rebane wrote:
>
> >
> >
> >
> >  Forwarded Message 
> > Subject: Re: At least one JAR was scanned for TLDs yet contained
> > no TLDs.
> > Date: Thu, 8 Oct 2020 13:37:49 +0300
> > From: Raivo Rebane 
> > To: Martin Grigorov ,
> > users-get.123_...@tomcat.apache.org
> >
> >
> >
> > Hello
> >
> > I debaged the jars list and got following:
> >
> > 08-Oct-2020 13:27:07.240 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-parser.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-transcoder.jar].
> > Consider adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-gui-util.jar].
> > Consider adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-svggen.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-xml.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.241 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-awt-util.jar].
> > Consider adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.242 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-util.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.242 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-script.jar]. Consider
> > adding the JAR to the
> > tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> > CATALINA_BASE/conf/catalina.properties file.
> > 08-Oct-2020 13:27:07.242 FINE [main]
> > org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> > files were found in
> > [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-brid

Re: Re: At least one JAR was scanned for TLDs yet contained no TLDs.

[Martin Grigorov]

On Thu, Oct 8, 2020, 14:00 Raivo Rebane  wrote:

>
>
>
>  Forwarded Message 
> Subject:Re: At least one JAR was scanned for TLDs yet contained no
> TLDs.
> Date:   Thu, 8 Oct 2020 13:37:49 +0300
> From:   Raivo Rebane 
> To: Martin Grigorov ,
> users-get.123_...@tomcat.apache.org
>
>
>
> Hello
>
> I debaged the jars list and got following:
>
> 08-Oct-2020 13:27:07.240 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-parser.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-transcoder.jar].
> Consider adding the JAR to the
> tomcat.util.scan.StandardJarScanFilter.jarsToSkip property in
> CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-gui-util.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-svggen.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-xml.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.241 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-awt-util.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-util.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-script.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-bridge.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-dom.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-gvt.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-swing.jar]. Consider
> adding the JAR to the tomcat.util.scan.StandardJarScanFilter.jarsToSkip
> property in CATALINA_BASE/conf/catalina.properties file.
> 08-Oct-2020 13:27:07.242 FINE [main]
> org.apache.jasper.servlet.TldScanner$TldScannerCallback.scan No TLD
> files were found in
> [file:/opt/tomcat/apache-tomcat-9.0.38/lib/batik-constants.jar].
> Consider adding the JAR to the
> tomcat.util.scan.StandardJarScanFilter.jarsT

Re: At least one JAR was scanned for TLDs yet contained no TLDs.

[Martin Grigorov]

Hi,

On Wed, Oct 7, 2020 at 9:46 PM Raivo Rebane  wrote:

> Hello
>
> If I start Tomcat9 I got following:
>
> [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR
> was scanned for TLDs yet contained no TLDs. Enable debug logging for
> this logger for a complete list of JARs that were scanned but no TLDs
> were found in them.
>
>   I would like to know which JARS is mentioned
>
> I add following lines to catalina.properies:
>
> org.apache.jasper.compiler.TldLocationsCache.level = ALL
> org.apache.jasper.servlet.TldScanner.level = ALL
>
> But I dont have list of these JARS
>
> And how to add these JARS to classpath anyway ?
>

Enable debug for org.apache.jasper.servlet.TldScanner


>
> Regards
>
> Raivo
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Is Tomcat7 supports HTTP2

[Martin Grigorov]

Hi,

On Thu, Oct 8, 2020 at 9:32 AM Tosh, Bibhuti Bhusan (Bibhuti) <
bt...@avaya.com> wrote:

> HI All,
> I am an user of tomcat7 version. I wanted to know this version tomcat
> 7.0.105 supports HTTP2 and CVE-2020-11996 is still applicable to tis
> version. I did not any reference of tomcat7 supporting HTTP2 and so asking
> this questions to community. Thanks in advance.
>

No, HTTP/2 is available in 8.5.+


>
> Regards,
> --Bibhuti
>
>

Re: VersionLoggerListener

[Martin Grigorov]

On Wed, Oct 7, 2020 at 2:19 PM Raivo Rebane  wrote:

> Hello
>
> I removed  org.apache.catalina.startup.VersionLoggerListener
>
> but I got following error:
>
> java.lang.NoSuchMethodError: 'org.apache.catalina.Engine
> org.apache.catalina.Service.getContainer()'
>

I'd recommend you to download a new .zip/.tgz, or re-install Tomcat.

Martin


> Regards
>
> Raivo
>
> On 07.10.20 14:01, Martin Grigorov wrote:
> > Hi,
> >
> > On Wed, Oct 7, 2020 at 1:51 PM Raivo Rebane  wrote:
> >
> >> Hello
> >>
> >> My tomcat9 don't starts and gives error:
> >>
> >>java.lang.ClassNotFoundException:
> >> org.apache.catalina.startup.VersionLoggerListener
> >>
> >> How to resolve this problem ?
> >>
> > Download Tomcat again. Your installation must be corrupted somehow.
> > Or remove org.apache.catalina.startup.VersionLoggerListener from
> > conf/server.xml
> >
> >
> >> Regards
> >>
> >> Raivo
> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
>

Re: VersionLoggerListener

[Martin Grigorov]

Hi,

On Wed, Oct 7, 2020 at 1:51 PM Raivo Rebane  wrote:

> Hello
>
> My tomcat9 don't starts and gives error:
>
>   java.lang.ClassNotFoundException:
> org.apache.catalina.startup.VersionLoggerListener
>
> How to resolve this problem ?
>

Download Tomcat again. Your installation must be corrupted somehow.
Or remove org.apache.catalina.startup.VersionLoggerListener from
conf/server.xml


>
> Regards
>
> Raivo
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 12:47 PM Mark Thomas  wrote:

> On 06/10/2020 10:35, Martin Grigorov wrote:
> > On Tue, Oct 6, 2020 at 12:15 PM Martin Grigorov 
> > wrote:
> >
> >>
> >>
> >> On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas  wrote:
> >>
> >>> On 06/10/2020 07:30, Martin Grigorov wrote:
> >>>> Hi,
> >>>>
> >>>> I face an issue with the NIO2 protocol.
> >>>> When I increase the request rate to more than 500 requests per second
> it
> >>>> starts failing with:
> >>>
> >>> I'm unable to reproduce this with a 10.0.x build requesting a simple
> >>> text file from the ROOT web app using NIO2 and Java 1.8.0_265.
> >>>
> >>> I see around 15k req/s and no failures.
> >>>
> >>
> >> I use latest 9.0.x.
> >> Let me try with non-embedded Tomcat!
> >>
> >
> > Same build of Tomcat started with ./bin/startup.sh has no such problems!
> > It seems
> >
> https://github.com/martin-g/http2-server-perf-tests/blob/master/java/tomcat/src/main/java/info/mgsolutions/tomcat/TomcatEmbedded.java
> > either misses something or does something more than what conf/server.xml
> > does.
> > I've tried to minify server.xml by removing all listeners, valves and
> JNDI
> > related stuff but it still works fine.
>
> maxThreads="8" setting? That looks very low to me.
>

Yep! This was the reason! Thanks!
The EOFs happen with standalone Tomcat too when maxThreads is 8.

I've reduced it because otherwise the throughput of NIO is lower. Async
Profiler (perf) says that there are many context switches.


>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 12:15 PM Martin Grigorov 
wrote:

>
>
> On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas  wrote:
>
>> On 06/10/2020 07:30, Martin Grigorov wrote:
>> > Hi,
>> >
>> > I face an issue with the NIO2 protocol.
>> > When I increase the request rate to more than 500 requests per second it
>> > starts failing with:
>>
>> I'm unable to reproduce this with a 10.0.x build requesting a simple
>> text file from the ROOT web app using NIO2 and Java 1.8.0_265.
>>
>> I see around 15k req/s and no failures.
>>
>
> I use latest 9.0.x.
> Let me try with non-embedded Tomcat!
>

Same build of Tomcat started with ./bin/startup.sh has no such problems!
It seems
https://github.com/martin-g/http2-server-perf-tests/blob/master/java/tomcat/src/main/java/info/mgsolutions/tomcat/TomcatEmbedded.java
either misses something or does something more than what conf/server.xml
does.
I've tried to minify server.xml by removing all listeners, valves and JNDI
related stuff but it still works fine.


>
>
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 12:13 PM Rémy Maucherat  wrote:

> On Tue, Oct 6, 2020 at 8:31 AM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > I face an issue with the NIO2 protocol.
> > When I increase the request rate to more than 500 requests per second it
> > starts failing with:
> >
> > 06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6]
> > org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State
> > [NEW]
> > 06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6]
> > org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed
> > Connection [1], Stream [0], Frame type [null], Error
> > java.io.EOFException
> > at
> > org.apache.tomcat.util.net
> > .SecureNio2Channel$2.completed(SecureNio2Channel.java:1005)
> >
>
> Also, it is a real EOF as the error says, caused by the client
> disconnecting for whatever reason (good or bad).
>

The client is exactly the same for Tomcat NIO/NIO2/APR, Golang, Node.js,
Rust and Netty.
It works for all but NIO2.


>
> Rémy
>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 12:06 PM Mark Thomas  wrote:

> On 06/10/2020 07:30, Martin Grigorov wrote:
> > Hi,
> >
> > I face an issue with the NIO2 protocol.
> > When I increase the request rate to more than 500 requests per second it
> > starts failing with:
>
> I'm unable to reproduce this with a 10.0.x build requesting a simple
> text file from the ROOT web app using NIO2 and Java 1.8.0_265.
>
> I see around 15k req/s and no failures.
>

I use latest 9.0.x.
Let me try with non-embedded Tomcat!


>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Unclear sentence in FailedRequestFilter's javadoc

[Martin Grigorov]

Hi Konstantin,

On Mon, Oct 5, 2020 at 2:53 PM Konstantin Kolinko 
wrote:

> пн, 5 окт. 2020 г. в 14:16, Martin Grigorov :
> >
> > Hi,
> >
> > What is the meaning of "addresses" at
> >
> https://github.com/apache/tomcat/blob/a0fb5f4b42c593fa566878dddfa213e5f3c9c499/java/org/apache/catalina/filters/FailedRequestFilter.java#L40
> > ?
> >
> > "Parameter parsing does check content type of the request, so there
> should
> > not be problems with *addresses* that use
> > request.getInputStream() and
> request.getReader()"
> >
> > I am not English native speaker, so it might be correct, but it looks
> like
> > the wrong word is used here.
>
> It looks odd for me as well, although from the commit history it looks
> that it was my commit that added that sentence [1], 9 years ago.
>
> Essentially, the text should be the same as in description of the
> filter in Configuration Reference [2][3]. It talks about "the servlet
> protected by this filter".
>
> [1] https://svn.apache.org/viewvc?view=revision=1198707
> [2] https://svn.apache.org/viewvc?view=revision=1200107
> [3]
> http://tomcat.apache.org/tomcat-9.0-doc/config/filter.html#Failed_Request_Filter


Thanks for the answer!
I've updated it to use the same text as at the website docs.


>
>
> It looks as if you are the first person who has read that Javadoc in 9
> years.
>

Yeah, usually people read the code, not the docs :-)

Cheers!
Martin


> Best regards,
> Konstantin Kolinko
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

On Tue, Oct 6, 2020 at 9:30 AM Martin Grigorov  wrote:

> Hi,
>
> I face an issue with the NIO2 protocol.
> When I increase the request rate to more than 500 requests per second it
> starts failing with:
>
> 06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6]
> org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State [NEW]
> 06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6]
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed
> Connection [1], Stream [0], Frame type [null], Error
> java.io.EOFException
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1005)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
> at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
> at
> java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568)
> at
> java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1126)
> at
> org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025)
> at
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
> at
> org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1054)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1112)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
> at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
> at
> java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568)
> at
> java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
> at
> java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1038)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
> at
> org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1124)
> at
> org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025)
> at
> org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.start(Nio2Endpoint.java:1004)
> at
> org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1450)
> at
> org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1293)
> at
> org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1265)
> at
> org.apache.coyote.http2.Http2AsyncParser.readConnectionPreface(Http2AsyncParser.java:55)
> at
> org.apache.coyote.http2.Http2UpgradeHandler.init(Http2UpgradeHandler.java:241)
> at
> org.apache.coyote.http2.Http2AsyncUpgradeHandler.init(Http2AsyncUpgradeHandler.java:40)
> at
> org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:316)
> at
> org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:40)
> at
> org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
> at
> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)
> at
> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
> at
> org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1679)
> at
> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
> at
> org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1104)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:99)
> at
> org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:92)
> at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
> at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)
> at
> java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> at
> java.base/java.util.concurrent.ThreadP

A lot of EOFs when using NIO2 with HTTP2

[Martin Grigorov]

Hi,

I face an issue with the NIO2 protocol.
When I increase the request rate to more than 500 requests per second it
starts failing with:

06-Oct-2020 09:18:19.964 FINE [https-jsse-nio2-8080-exec-6]
org.apache.coyote.http2.Http2UpgradeHandler.init Connection [1], State [NEW]
06-Oct-2020 09:18:19.978 FINE [https-jsse-nio2-8080-exec-6]
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.failed
Connection [1], Stream [0], Frame type [null], Error
java.io.EOFException
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1005)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
at
java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568)
at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
at
org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1126)
at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1054)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1112)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
at java.base/sun.nio.ch.Invoker.invokeDirect(Invoker.java:158)
at
java.base/sun.nio.ch.UnixAsynchronousSocketChannelImpl.implRead(UnixAsynchronousSocketChannelImpl.java:568)
at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:276)
at
java.base/sun.nio.ch.AsynchronousSocketChannelImpl.read(AsynchronousSocketChannelImpl.java:297)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1038)
at
org.apache.tomcat.util.net.SecureNio2Channel$2.completed(SecureNio2Channel.java:1001)
at
org.apache.tomcat.util.net.SecureNio2Channel.read(SecureNio2Channel.java:1124)
at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.run(Nio2Endpoint.java:1025)
at
org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$Nio2OperationState.start(Nio2Endpoint.java:1004)
at
org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1450)
at
org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1293)
at
org.apache.tomcat.util.net.SocketWrapperBase.read(SocketWrapperBase.java:1265)
at
org.apache.coyote.http2.Http2AsyncParser.readConnectionPreface(Http2AsyncParser.java:55)
at
org.apache.coyote.http2.Http2UpgradeHandler.init(Http2UpgradeHandler.java:241)
at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.init(Http2AsyncUpgradeHandler.java:40)
at
org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch(Http2UpgradeHandler.java:316)
at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.upgradeDispatch(Http2AsyncUpgradeHandler.java:40)
at
org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at
org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1679)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at
org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1104)
at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:99)
at
org.apache.tomcat.util.net.SecureNio2Channel$HandshakeReadCompletionHandler.completed(SecureNio2Channel.java:92)
at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)
at
java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:832)
06-Oct-2020 09:18:19.980 FINE [https-jsse-nio2-8080-exec-6]
org.apache.coyote.http2.Http2UpgradeHandler.upgradeDispatch Entry,
Connection [1], SocketStatus [ERROR]

If I just change the protocol to org.apache.coyote.http11.Http11NioProtocol
then everything is OK.

To reproduce:
1) git clone https://github.com/martin-g/http2-server-perf-tests.git
2

Unclear sentence in FailedRequestFilter's javadoc

[Martin Grigorov]

Hi,

What is the meaning of "addresses" at
https://github.com/apache/tomcat/blob/a0fb5f4b42c593fa566878dddfa213e5f3c9c499/java/org/apache/catalina/filters/FailedRequestFilter.java#L40
?

"Parameter parsing does check content type of the request, so there should
not be problems with *addresses* that use
request.getInputStream() and request.getReader()"

I am not English native speaker, so it might be correct, but it looks like
the wrong word is used here.

Martin

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

On Thu, Oct 1, 2020 at 12:29 PM Mark Thomas  wrote:

> On 30/09/2020 18:47, Martin Grigorov wrote:
> > On Wed, Sep 30, 2020 at 7:47 PM Mark Thomas  wrote:
> >> On 30/09/2020 16:17, Mark Thomas wrote:
>
> 
>
> >>> That is helpful. Looks like you have found a way to reproduce the
> buffer
> >>> issues reported in
> https://bz.apache.org/bugzilla/show_bug.cgi?id=64710
> >>
> >> Can you share the command you used to trigger those errors please.
> >>
> >
> > The Vegeta command I used is:
> >
> > jq -ncM '{"method": "POST", "url": "
> https://localhost:8080/testbed/plaintext;,
> > "body":"payload=Some
> >
> sdgggwwsdgssfshffheeepayload"
> > | @base64, header: {"Content-Type":
> > ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
> > -http2 -rate=1000 -max-workers=8 -insecure -duration=2m | vegeta encode >
> > /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
> >
> > The app is at
> >
> https://github.com/martin-g/http2-server-perf-tests/tree/master/java/tomcat
> .
> > Just start EmbeddedTomcat#main() with -Dtomcat.http2=true
>
> Definitely timing related as I am unable to reproduce the problem with
> that command or some variations.
>
> However, I think I have managed to track down the root cause. The good
> news is that the BufferOverflowException is largely harmless. It is a
> side-effect of the connection being closed due to an error. My guess is
> that the error was a combination of vegeta sending an unexpected reset
> frame and Tomcat maintaining state for a very small number of streams in
> some circumstances.
>
> If you could retest with the latest 9.0.x that would be very helpful.
> The memory usage, stream state maintenance and this
> BufferOverflowException should all be fixed.
>

Yesterday it was very easy to reproduce it here.
It looks good now - both exception types didn't happen in several runs!

But something new broke:


SEVERE: Servlet.service() for servlet [plaintext] in context with path []
threw exception
java.lang.NullPointerException: Cannot throw exception because "ioe" is null
at
org.apache.coyote.http2.Http2UpgradeHandler.handleAppInitiatedIOException(Http2UpgradeHandler.java:797)
at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.handleAsyncException(Http2AsyncUpgradeHandler.java:276)
at
org.apache.coyote.http2.Http2AsyncUpgradeHandler.writeWindowUpdate(Http2AsyncUpgradeHandler.java:252)
at org.apache.coyote.http2.Stream$StreamInputBuffer.doRead(Stream.java:1088)
at org.apache.coyote.Request.doRead(Request.java:555)
at
org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:336)
at
org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:632)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:362)
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:132)
at org.apache.catalina.connector.Request.readPostBody(Request.java:3308)
at org.apache.catalina.connector.Request.parseParameters(Request.java:3241)
at org.apache.catalina.connector.Request.getParameter(Request.java:1124)
at
org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:381)
at info.mgsolutions.tomcat.PlainTextServlet.doPost(PlainTextServlet.java:41)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

I will improve it!


> Mark
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

On Wed, Sep 30, 2020 at 7:47 PM Mark Thomas  wrote:

> On 30/09/2020 16:17, Mark Thomas wrote:
> > On 30/09/2020 13:53, Martin Grigorov wrote:
> >> On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov 
> >
> >
> > 
> >
> >> When I load test HTTP2 with POST (with big bodies) there are many errors
> >> like:
> >>
> >> 1)
> >> Exception in thread "https-jsse-nio-8080-exec-5"
> >> java.nio.BufferOverflowException
> >> at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957)
> >> at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247)
> >> at
> >> org.apache.tomcat.util.net
> .SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
> >> at
> >> org.apache.tomcat.util.net
> .SocketWrapperBase.unRead(SocketWrapperBase.java:401)
> >> at
> >>
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307)
> >> at
> >>
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164)
> >> at
> >> org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
> >> at
> >> org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> >> at
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> >> at
> >>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
> >> at
> >>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> >> at java.base/java.lang.Thread.run(Thread.java:832)
> >>
> >> 2)
> >> Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller
> events
> >> SEVERE: Failed to register socket with selector from poller
> >> java.nio.channels.ClosedChannelException
> >> at
> >>
> java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222)
> >> at
> >> org.apache.tomcat.util.net
> .NioEndpoint$Poller.events(NioEndpoint.java:609)
> >> at org.apache.tomcat.util.net
> .NioEndpoint$Poller.run(NioEndpoint.java:703)
> >> at java.base/java.lang.Thread.run(Thread.java:832)
> >
> > That is helpful. Looks like you have found a way to reproduce the buffer
> > issues reported in https://bz.apache.org/bugzilla/show_bug.cgi?id=64710
>
> Can you share the command you used to trigger those errors please.
>

The Vegeta command I used is:

jq -ncM '{"method": "POST", "url": "https://localhost:8080/testbed/plaintext;,
"body":"payload=Some
sdgggwwsdgssfshffheeeeeeeeeeepayload"
| @base64, header: {"Content-Type":
["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
-http2 -rate=1000 -max-workers=8 -insecure -duration=2m | vegeta encode >
/tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .

The app is at
https://github.com/martin-g/http2-server-perf-tests/tree/master/java/tomcat.
Just start EmbeddedTomcat#main() with -Dtomcat.http2=true

Martin


>
> Thanks,
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

On Wed, Sep 30, 2020 at 12:50 PM Martin Grigorov 
wrote:

> Hi,
>
> On Wed, Sep 30, 2020 at 11:35 AM Mark Thomas  wrote:
>
>> On 30/09/2020 06:42, Arshiya Shariff wrote:
>> > Hi Martin ,
>> >
>> > Thank you for the response.
>> >
>> > With a payload of 200 bytes we were able to send 20K requests/sec with
>> 200 users from Jmeter without any memory issue . On increasing the payload
>> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
>> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
>> memory is cleared in the G1 mixed GC itself , but with 1000 users the
>> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
>> seconds to clear the memory. These cases were executed with maxThreads 200
>> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
>> still GC was struggling .
>> >
>> > When we tried with 10 instances of JMeter , each with 100 users , where
>> each instance was started with a delay of 1 minute we were able to see 1000
>> connections created in tomcat without any memory issues. But when 1000
>> users are created using single instance of JMeter in 20 seconds , tomcat's
>> memory is filling fast- 20GB in 2 minutes.
>> > We suspect that the burst of connections being opened has a problem .
>> Please help us with the same .
>> >
>> > On analyzing the heap dump we see
>> org.apache.tomcat.util.collections.SynchronizedStack occupying around 93%
>> of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump.
>>
>> You can't have high throughput, low GC pauses and small heap sizes.
>> Broadly you can have any two of those three at the expense of the third.
>>
>> The way Tomcat currently retains information about completed h2 streams
>> means you are likely to need a large heap under heavy load. There are
>> some changes already in 10.0.x that I plan to back-port to 9.0.x and
>> 8.5.x later today that should significantly reduce the heap requirements.
>>
>
> Here is a screenshot of me loading Tomcat HTTP2 9.0.x+the changes from
> 10.0.x with Vegeta for 3 mins:
> https://pasteboard.co/JtshrAs.png
> As you can see the GC is properly cleaning the heap. At the end the memory
> is not released until the GC kicks.
>
> Note: this is with a GET request without a body! I'm going to start a new
> email thread for POST with body - there I get GOAWAY+ENHANCE_YOUR_CALM for
> very low load.
>

When I load test HTTP2 with POST (with big bodies) there are many errors
like:

1)
Exception in thread "https-jsse-nio-8080-exec-5"
java.nio.BufferOverflowException
at java.base/java.nio.ByteBuffer.put(ByteBuffer.java:957)
at java.base/java.nio.HeapByteBuffer.put(HeapByteBuffer.java:247)
at
org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
at
org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401)
at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:307)
at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:164)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:832)

2)
Sep 30, 2020 3:44:04 PM org.apache.tomcat.util.net.NioEndpoint$Poller events
SEVERE: Failed to register socket with selector from poller
java.nio.channels.ClosedChannelException
at
java.base/java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:222)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:609)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:703)
at java.base/java.lang.Thread.run(Thread.java:832)

And the client gets these kind of error responses:

"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=7, ErrCode=PROTOCOL_ERROR,
debug=\"Connection [9354], Stream [7], The content length header value
[255] does not agree with the size of the data received [0]\"",
"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=31,
ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [9355], Too much 

Re: HTTP2: POST request with a small body leads to GOAWAY + ErrCode=ENHANCE_YOUR_CALM

[Martin Grigorov]

On Wed, Sep 30, 2020 at 1:15 PM Martin Grigorov 
wrote:

>
>
> On Wed, Sep 30, 2020 at 1:01 PM Martin Grigorov 
> wrote:
>
>> Hi,
>>
>> I've tried to test the scenario of Arshiya Shariff:
>> "With a payload of 200 bytes we were able to send 20K requests/sec with
>> 200 users from Jmeter without any memory issue . On increasing the payload
>> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
>> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
>> memory is cleared in the G1 mixed GC itself , but with 1000 users the
>> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
>> seconds to clear the memory. These cases were executed with maxThreads 200
>> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
>> still GC was struggling ."
>>
>> My Servlet looks like this:
>> https://github.com/martin-g/http2-server-perf-tests/blob/f5e73d73a864d4c7dbcbbbe80024764352383116/java/tomcat/src/main/java/info/mgsolutions/tomcat/PlainTextServlet.java#L37-L54
>>
>> And the Vegeta command I issue is:
>>
>> jq -ncM '{"method": "POST", "url": "
>> https://localhost:8080/testbed/plaintext;, "body":"payload=Some payload"
>> | @base64, header: {"Content-Type":
>> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
>> -http2 -rate=1 -max-workers=8 -insecure -duration=3s | vegeta encode >
>> /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
>>
>> The outcome is:
>>
>> {
>>   "latencies": {
>> "total": 114369584,
>> "mean": 38123194,
>> "50th": 45737841,
>> "90th": 62258803,
>> "95th": 62258803,
>> "99th": 62258803,
>> "max": 62258803,
>> "min": 6372940
>>   },
>>   "bytes_in": {
>> "total": 24,
>> "mean": 8
>>   },
>>   "bytes_out": {
>> "total": 40,
>> "mean": 13.334
>>   },
>>   "earliest": "2020-09-30T12:40:15.208111926+03:00",
>>   "latest": "2020-09-30T12:40:17.208170426+03:00",
>>   "end": "2020-09-30T12:40:17.253908267+03:00",
>>   "duration": 258500,
>>   "wait": 45737841,
>>   "requests": 3,
>>   "rate": 1.499956126283306,
>>   "throughput": 0.9776144183650195,
>>   "success": 0.,
>>   "status_codes": {
>> "0": 1,
>> "200": 2
>>   },
>>   "errors": [
>> "Post \"https://localhost:8080/testbed/plaintext\": http2: server
>> sent GOAWAY and closed the connection; LastStreamID=3,
>> ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [4], Too much overhead so the
>> connection will be closed\""
>>   ]
>> }
>>
>> I.e. it sends 1 POST request every second for 3 seconds and this leads to
>> GOAWAY+ENHANCE_YOUR_CALM !
>>
>> From my understanding on "overhead" this should happen if there are more
>> SETTINGS|PING|WINDOW_UPDATE|PRIORITY frames than HTTP_HEADERS or DATA.
>>
>> I am not able to decrypt TLS data in Wireshark when the client is Vegeta,
>> so I will try to debug it or to reproduce it with Firefox (to be able to
>> use Wireframe).
>>
>> Any ideas why this happens ?
>>
>
> The reason is here:
> https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L1448
> overheadThreshold is 1024 and average is 20
> It goes twice in this method and then fails at
> https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L352
>

https://bz.apache.org/bugzilla/show_bug.cgi?id=63690#c7 explains the issue.
No idea why but Vegeta/Golang sends two DATA frames and the second one
(with END_STREAM) is empty:

30-Sep-2020 14:45:35.646 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Stream.emitHeader Connection [1], Stream [1], HTTP
header [accept-encoding], Value [gzip]
30-Sep-2020 14:45:35.646 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Stream.emitHeader Connection [1], Stream [1], HTTP
header [user-agent], Value [Go-http-client/2.0]
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.swallow Connection [1], Stream [1],
Swallowed [0] bytes
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.validateFrame Connection [1], Stream
[1], Frame type [DATA], Flags [0], Payload size [20]
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.readDataFrame Connection [1], Stream
[1], Data length, [20], Padding length [none]
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Stream$StreamInputBuffer.onDataAvailable Data added
to inBuffer when read thread is waiting. Signalling that thread to continue
30-Sep-2020 14:45:35.647 FINE [https-jsse-nio-8080-exec-6]
org.apache.coyote.http2.Http2Parser.validateFrame Connection [1], Stream
[1], Frame type [DATA], Flags [1], Payload size [0]



>
>
>>
>> Martin
>>
>

Re: HTTP2: POST request with a small body leads to GOAWAY + ErrCode=ENHANCE_YOUR_CALM

[Martin Grigorov]

On Wed, Sep 30, 2020 at 1:01 PM Martin Grigorov 
wrote:

> Hi,
>
> I've tried to test the scenario of Arshiya Shariff:
> "With a payload of 200 bytes we were able to send 20K requests/sec with
> 200 users from Jmeter without any memory issue . On increasing the payload
> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
> memory is cleared in the G1 mixed GC itself , but with 1000 users the
> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
> seconds to clear the memory. These cases were executed with maxThreads 200
> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
> still GC was struggling ."
>
> My Servlet looks like this:
> https://github.com/martin-g/http2-server-perf-tests/blob/f5e73d73a864d4c7dbcbbbe80024764352383116/java/tomcat/src/main/java/info/mgsolutions/tomcat/PlainTextServlet.java#L37-L54
>
> And the Vegeta command I issue is:
>
> jq -ncM '{"method": "POST", "url": "
> https://localhost:8080/testbed/plaintext;, "body":"payload=Some payload"
> | @base64, header: {"Content-Type":
> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
> -http2 -rate=1 -max-workers=8 -insecure -duration=3s | vegeta encode >
> /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
>
> The outcome is:
>
> {
>   "latencies": {
> "total": 114369584,
> "mean": 38123194,
> "50th": 45737841,
> "90th": 62258803,
> "95th": 62258803,
> "99th": 62258803,
> "max": 62258803,
> "min": 6372940
>   },
>   "bytes_in": {
> "total": 24,
> "mean": 8
>   },
>   "bytes_out": {
> "total": 40,
> "mean": 13.334
>   },
>   "earliest": "2020-09-30T12:40:15.208111926+03:00",
>   "latest": "2020-09-30T12:40:17.208170426+03:00",
>   "end": "2020-09-30T12:40:17.253908267+03:00",
>   "duration": 258500,
>   "wait": 45737841,
>   "requests": 3,
>   "rate": 1.499956126283306,
>   "throughput": 0.9776144183650195,
>   "success": 0.,
>   "status_codes": {
> "0": 1,
> "200": 2
>   },
>   "errors": [
> "Post \"https://localhost:8080/testbed/plaintext\": http2: server
> sent GOAWAY and closed the connection; LastStreamID=3,
> ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [4], Too much overhead so the
> connection will be closed\""
>   ]
> }
>
> I.e. it sends 1 POST request every second for 3 seconds and this leads to
> GOAWAY+ENHANCE_YOUR_CALM !
>
> From my understanding on "overhead" this should happen if there are more
> SETTINGS|PING|WINDOW_UPDATE|PRIORITY frames than HTTP_HEADERS or DATA.
>
> I am not able to decrypt TLS data in Wireshark when the client is Vegeta,
> so I will try to debug it or to reproduce it with Firefox (to be able to
> use Wireframe).
>
> Any ideas why this happens ?
>

The reason is here:
https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L1448
overheadThreshold is 1024 and average is 20
It goes twice in this method and then fails at
https://github.com/apache/tomcat/blob/16181fc7b1930ff202ec2e475f2fbdc587f3e314/java/org/apache/coyote/http2/Http2UpgradeHandler.java#L352


>
> Martin
>

HTTP2: POST request with a small body leads to GOAWAY + ErrCode=ENHANCE_YOUR_CALM

[Martin Grigorov]

Hi,

I've tried to test the scenario of Arshiya Shariff:
"With a payload of 200 bytes we were able to send 20K requests/sec with 200
users from Jmeter without any memory issue . On increasing the payload to
5Kb and the number of users to 1000 in Jmeter and sending 1000 requests per
second , the heap of 20GB got filled in 2 minutes . With 200 users the
memory is cleared in the G1 mixed GC itself , but with 1000 users the
memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
seconds to clear the memory. These cases were executed with maxThreads 200
in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
still GC was struggling ."

My Servlet looks like this:
https://github.com/martin-g/http2-server-perf-tests/blob/f5e73d73a864d4c7dbcbbbe80024764352383116/java/tomcat/src/main/java/info/mgsolutions/tomcat/PlainTextServlet.java#L37-L54

And the Vegeta command I issue is:

jq -ncM '{"method": "POST", "url": "https://localhost:8080/testbed/plaintext;,
"body":"payload=Some payload" | @base64, header: {"Content-Type":
["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
-http2 -rate=1 -max-workers=8 -insecure -duration=3s | vegeta encode >
/tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .

The outcome is:

{
  "latencies": {
"total": 114369584,
"mean": 38123194,
"50th": 45737841,
"90th": 62258803,
"95th": 62258803,
"99th": 62258803,
"max": 62258803,
"min": 6372940
  },
  "bytes_in": {
"total": 24,
"mean": 8
  },
  "bytes_out": {
"total": 40,
"mean": 13.334
  },
  "earliest": "2020-09-30T12:40:15.208111926+03:00",
  "latest": "2020-09-30T12:40:17.208170426+03:00",
  "end": "2020-09-30T12:40:17.253908267+03:00",
  "duration": 258500,
  "wait": 45737841,
  "requests": 3,
  "rate": 1.499956126283306,
  "throughput": 0.9776144183650195,
  "success": 0.,
  "status_codes": {
"0": 1,
"200": 2
  },
  "errors": [
"Post \"https://localhost:8080/testbed/plaintext\": http2: server sent
GOAWAY and closed the connection; LastStreamID=3,
ErrCode=ENHANCE_YOUR_CALM, debug=\"Connection [4], Too much overhead so the
connection will be closed\""
  ]
}

I.e. it sends 1 POST request every second for 3 seconds and this leads to
GOAWAY+ENHANCE_YOUR_CALM !

>From my understanding on "overhead" this should happen if there are more
SETTINGS|PING|WINDOW_UPDATE|PRIORITY frames than HTTP_HEADERS or DATA.

I am not able to decrypt TLS data in Wireshark when the client is Vegeta,
so I will try to debug it or to reproduce it with Firefox (to be able to
use Wireframe).

Any ideas why this happens ?

Martin

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

Hi,

On Wed, Sep 30, 2020 at 11:35 AM Mark Thomas  wrote:

> On 30/09/2020 06:42, Arshiya Shariff wrote:
> > Hi Martin ,
> >
> > Thank you for the response.
> >
> > With a payload of 200 bytes we were able to send 20K requests/sec with
> 200 users from Jmeter without any memory issue . On increasing the payload
> to 5Kb and the number of users to 1000 in Jmeter and sending 1000 requests
> per second , the heap of 20GB got filled in 2 minutes . With 200 users the
> memory is cleared in the G1 mixed GC itself , but with 1000 users the
> memory is not cleared in the mixed GC , it takes full GCs of 7 to 10
> seconds to clear the memory. These cases were executed with maxThreads 200
> in tomcat , so we tried increasing the maxThreads from 200 to 1000, but
> still GC was struggling .
> >
> > When we tried with 10 instances of JMeter , each with 100 users , where
> each instance was started with a delay of 1 minute we were able to see 1000
> connections created in tomcat without any memory issues. But when 1000
> users are created using single instance of JMeter in 20 seconds , tomcat's
> memory is filling fast- 20GB in 2 minutes.
> > We suspect that the burst of connections being opened has a problem .
> Please help us with the same .
> >
> > On analyzing the heap dump we see
> org.apache.tomcat.util.collections.SynchronizedStack occupying around 93%
> of 3GB live data ,the remaining 17GB is Garbage collected in the heap dump.
>
> You can't have high throughput, low GC pauses and small heap sizes.
> Broadly you can have any two of those three at the expense of the third.
>
> The way Tomcat currently retains information about completed h2 streams
> means you are likely to need a large heap under heavy load. There are
> some changes already in 10.0.x that I plan to back-port to 9.0.x and
> 8.5.x later today that should significantly reduce the heap requirements.
>

Here is a screenshot of me loading Tomcat HTTP2 9.0.x+the changes from
10.0.x with Vegeta for 3 mins:
https://pasteboard.co/JtshrAs.png
As you can see the GC is properly cleaning the heap. At the end the memory
is not released until the GC kicks.

Note: this is with a GET request without a body! I'm going to start a new
email thread for POST with body - there I get GOAWAY+ENHANCE_YOUR_CALM for
very low load.

Martin


>
> Mark
>
>
> >
> > Thanks and Regards
> > Arshiya Shariff
> >
> > -Original Message-
> > From: Martin Grigorov 
> > Sent: Monday, September 28, 2020 11:44 PM
> > To: Tomcat Users List 
> > Subject: Re: HTTP2: memory filled up fast on increasing the connections
> to 1000/2000 (Embedded tomcat 9.0.38)
> >
> > Hi Arshiya,
> >
> >
> > On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff <
> arshiya.shar...@ericsson.com.invalid> wrote:
> >
> >> Hi All,
> >> With 200 threads(users) , ramp up duration of 2 seconds , loop count
> >> 80 and by sending 1000 http2 requests/sec from JMeter Client to an
> >> embedded tomcat application we did not observe any memory issue , but
> >> on sending
> >> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the
> >> application's heap space of 20 GB is occupied in 2 minutes and after 2
> >> full GCs the memory clears and comes down to 4GB (expected) .
> >>
> >
> > I am not sure whether you follow the other discussions at users@.
> > In another email thread we discuss load testing Tomcat HTTP2 and we are
> able to make around 12K reqs/s with another load testing tool -
> https://protect2.fireeye.com/v1/url?k=f8cfc13c-a66f0379-f8cf81a7-8692dc8284cb-2c0aae53194b790f=1=6a9c569d-7da1-4394-a9ac-bf72724992fa=https%3A%2F%2Fgithub.com%2Ftsenart%2Fvegeta
> > For me JMeter itself failed with OOM when increasing the number of the
> virtual users above 2K.
> > There are several improvements in Tomcat master and 9.0.x in the HTTP2
> area. Some of the changes are not yet downported to 9.0.x. We still test
> them, trying to avoid introducing regressions in 9.0.x.
> >
> >
> >>
> >> Embedded tomcat Version:9.0.38
> >> Max Threads : 200
> >>
> >
> > The number of threads should be less if you do only CPU calculations
> without IO/network. If your app blocks on IO/network calls then you need
> more spare threads.
> > With more threads there will be more context switches and less
> throughput.
> > That's why there is no one golden rule that applies to all applications.
> > 200 is a good default that works for most of the applications. But you
> need to test with different values to see which one gives the best
> performance for your scena

Re: ApacheCon @ Home Tomcat Track Schedule

[Martin Grigorov]

Hi,

A friendly reminder: the conference already started and the Tomcat sessions
will start in less than 6 hours from now!

On Fri, Aug 14, 2020 at 6:49 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> All,
>
> I'm happy to announce that the Apache Tomcat track schedule has been
> posted for ApacheCon @ Home, our virtual conference to replace
> "ApacheCon North America 2020". If you use social media to discuss
> this event, please use #ACAH2020 and tag @apachecon.
>
> You can find the schedule here, as well as links to the other 27 (!)
> tracks that will be held at the conference:
> https://www.apachecon.com/acah2020/tracks/tomcat.html
>
> If you'd like to register (zero cost!), you can do that here:
> https://hopin.to/events/apachecon-home
>
> Note that some tracks are happening at different times; please check
> the times for each presentation in each track for your timezone.
>
> I hope to "see" many of you at the conference this year.
>
> Thanks,
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl82sk4ACgkQHPApP6U8
> pFiypw/+M6Sz1wUtXk6A1CO5Lbz0jMU6/Ky53+zCb3skp4wNSVXaUuJ4xCb2hWg/
> cl8KkgP6N0e4stwB9M/XbL403Ic6NGYBvS2ZcMmm2rJqO99t5UvTpUY/DzzNwaXf
> d5z2h8p+aSU4ph6UZavi/Tydxa83WxQSRDuRww61xs6yHv7myELicv+fATJGY4fb
> slMLJt2Hzmh9C7t7+xkQDTpEUNz+oIV2yyQwuvkAS6WMHJbrqAkuqgQq6eAYm5Qc
> 79pb4e/G+E71Sr2AzeuFSpLQCUzYIyOkXFoLfr9L6Wb7iwwiu5bFP7+WAbJvI200
> tYjokn9NdZOwf4P3uY0xfCk3aq2jI9DZAcYjh7n0nhHCN+8vrqodQ/CZtoBYNQR5
> /HwMgszCyimfkegwb3CcuayFUko/Oa+klSOuh9XqFM8jb9XobyXx7dprZk3Ly0Kq
> oL/7h3+uBvz4rzR6FVdcuJ7stY4JpBhG+ZsjkiMktYcZovD9IP20RIE+omvqc0/S
> L1d/3XyRk3fM6JuQq2Y9yqTcgsbBrzEtcwIAOv2r9TQaRrwSbO2nTUiPVx8RTwQ4
> dtE60qVUvZ933I8aV4akqGkeUSBbxiv0WSfpV3T2F1latoqfEmlzueJNWrdgwXFC
> 8hSVyx29AsTiDGAXAswzxQDyNFlaY9zZZKi467doMvhJBGzTYfk=
> =pleB
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)

[Martin Grigorov]

Hi Arshiya,


On Mon, Sep 28, 2020 at 7:59 PM Arshiya Shariff
 wrote:

> Hi All,
> With 200 threads(users) , ramp up duration of 2 seconds , loop count 80
> and by sending 1000 http2 requests/sec from JMeter Client to an embedded
> tomcat application we did not observe any memory issue , but on sending
> 1000 http2 requests/sec with 2000 or 1000 users from JMeter , the
> application's heap space of 20 GB is occupied in 2 minutes and after 2 full
> GCs the memory clears and comes down to 4GB (expected) .
>

I am not sure whether you follow the other discussions at users@.
In another email thread we discuss load testing Tomcat HTTP2 and we are
able to make around 12K reqs/s with another load testing tool -
https://github.com/tsenart/vegeta
For me JMeter itself failed with OOM when increasing the number of the
virtual users above 2K.
There are several improvements in Tomcat master and 9.0.x in the HTTP2
area. Some of the changes are not yet downported to 9.0.x. We still test
them, trying to avoid introducing regressions in 9.0.x.


>
> Embedded tomcat Version:9.0.38
> Max Threads : 200
>

The number of threads should be less if you do only CPU calculations
without IO/network. If your app blocks on IO/network calls then you need
more spare threads.
With more threads there will be more context switches and less throughput.
That's why there is no one golden rule that applies to all applications.
200 is a good default that works for most of the applications. But you need
to test with different values to see which one gives the best performance
for your scenaria.


> All other properties are the tomcat defaults.
>
> Why is tomcat not able to process many connections ?
>

You can tell us by enabling -XX:+HeapDumpOnOutOfMemoryError and
-XX:HeapDumpPath=. Once you have the .hprof file you can
examine it with Eclipse Memory Analyzer tool and see what is leaking.
I will try to reproduce this issue tomorrow with Vegeta.


> Why is the memory filled when the connections are increased, are there any
> parameters to tune connections ?
> Please let us know.
>
> Thanks and Regards
> Arshiya Shariff
>

Re: ByteBuffer pooling

[Martin Grigorov]

On Mon, Sep 28, 2020 at 6:11 PM Rémy Maucherat  wrote:

> On Mon, Sep 28, 2020 at 4:34 PM Martin Grigorov 
> wrote:
>
> > Hi,
> >
> > I've profiled the memory allocation during load testing HTTP2:
> > https://pasteboard.co/Jtblqfl.png
> >
> > As you can see there are a lot of ByteBuffer.allocate(int) calls.
> > org.apache.catalina.connector.Response#Response(int)
> > org.apache.catalina.connector.Request#inputBuffer
> > org.apache.coyote.http2.Http2AsyncParser#readFrame
> > org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer
> >
> > Has it been discussed in the past to use a pool of ByteBuffer instances ?
> > Netty provides such for its abstraction ByteBuf:
> > https://netty.io/wiki/using-as-a-generic-library.html
> >
> > Here is a simple implementation:
> >
> >
> https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44
> > It's give() method could be optimized to not return the buffer to the
> pool
> > if the pool size is bigger than N, so it doesn't use huge list of buffers
> > which are used just once.
> >
> > The big unknown to me is: where to return the buffers to the pool ?
> > For HTTP2 maybe this could be done after writing the buffer to the
> socket.
> > If the Request/Response is recycled then their Input/OutputBuffer are
> > reused and everything is OK. But if recycling is not in use then new
> > allocations are done for each new request.
> >
> > I see that org.apache.tomcat.util.net.WriteBuffer does some pooling
> > already.
> >
> > What do you think?
> >
>
> I removed a lot of pooling in the connector with zero impact, so I don't
> see the point.
>

ConcurrentDateFormat pools SimpleDateFormat instances and it does make a
difference:
https://medium.com/@martin.grigorov/compare-performance-of-javas-simpledateformat-against-datetimeformatter-31be58cadf1d
 !
I'd give it a try for the HTTP2 Stream related allocations and let you know
how it goes!


>
> Rémy
>
>
> >
> > Martin
> >
>

ByteBuffer pooling

[Martin Grigorov]

Hi,

I've profiled the memory allocation during load testing HTTP2:
https://pasteboard.co/Jtblqfl.png

As you can see there are a lot of ByteBuffer.allocate(int) calls.
org.apache.catalina.connector.Response#Response(int)
org.apache.catalina.connector.Request#inputBuffer
org.apache.coyote.http2.Http2AsyncParser#readFrame
org.apache.coyote.http2.Stream.StreamOutputBuffer#buffer

Has it been discussed in the past to use a pool of ByteBuffer instances ?
Netty provides such for its abstraction ByteBuf:
https://netty.io/wiki/using-as-a-generic-library.html

Here is a simple implementation:
https://github.com/jhg023/Pbbl/blob/39c749b9e65f4f8a840a07812559cf8830bd5eae/src/main/java/com/github/pbbl/AbstractBufferPool.java#L44
It's give() method could be optimized to not return the buffer to the pool
if the pool size is bigger than N, so it doesn't use huge list of buffers
which are used just once.

The big unknown to me is: where to return the buffers to the pool ?
For HTTP2 maybe this could be done after writing the buffer to the socket.
If the Request/Response is recycled then their Input/OutputBuffer are
reused and everything is OK. But if recycling is not in use then new
allocations are done for each new request.

I see that org.apache.tomcat.util.net.WriteBuffer does some pooling already.

What do you think?

Martin

Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR

[Martin Grigorov]

On Fri, Sep 25, 2020 at 1:18 PM Arshiya Shariff
 wrote:

> Thanks Martin .
>
> Our expectation Is that , on receiving a RST_STREAM with CANCEL or
> NO_ERROR from the client after 1.5 secs for a particular stream , we don’t
> want the connection to be closed  with a GOAWAY:NO_ERROR  (while trying to
> send response after 2secs asynchronously over a stream that is closed) as
> the other streams during the same time are processing data fine .
>
> Additional info for GOAWAY: Connection [], Stream[],An error occurred
> during processing that was fatal to the connection .
>

I guess some other error happened and that led to the GOAWAY. But I cannot
be sure.
Any errors in the logs ?


>
> Thanks and Regards
> Arshiya Shariff
>
> -Original Message-
> From: Martin Grigorov 
> Sent: Tuesday, September 22, 2020 7:31 PM
> To: Tomcat Users List 
> Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a
> stream where the client has already sent RST_STREAM:NO ERROR
>
> Hi,
>
> On Tue, Sep 22, 2020 at 1:47 PM Arshiya Shariff <
> arshiya.shar...@ericsson.com.invalid> wrote:
>
> > Thank you so much Martin for the response.
> >  Yes , 9.0.38 testing is on going .
> >
> > As we don’t get this clear with the RFC , please help us with the
> > below two cases :
>
>   * If a client sends RST_STREAM with NO_ERROR, then while sending async
> > response is it expected behavior to close connection with GOAWAY ?
> >   * If a client sends RST_STREAM with CANCEL , then while sending
> > async response will tomcat send RST_STREAM or GOAWAY , from http2
> > protocol perspective ?
> >
>
> As per https://tools.ietf.org/html/rfc7540#section-6.4 and
> https://tools.ietf.org/html/rfc7540#section-5.1 if "send R" or "recv R"
> are processed then the stream is moved to CLOSED state.
> GOAWAY should be sent to the client only if the connection will be closed
> ( https://tools.ietf.org/html/rfc7540#section-6.8). The server should
> close the connection only if some serious problem has happened, e.g. an
> IOException.
>
> Tell us more about your use case. What do you want to do when 1.5secs pass
> ? What do you expect to happen ?
>
>
> >
> > Thanks and Regards
> > Arshiya Sharif
> >
> > -Original Message-
> > From: Martin Grigorov 
> > Sent: Tuesday, September 22, 2020 1:18 AM
> > To: Tomcat Users List 
> > Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a
> > stream where the client has already sent RST_STREAM:NO ERROR
> >
> > Hi,
> >
> > On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff <
> > arshiya.shar...@ericsson.com.invalid> wrote:
> >
> > > Hi All,
> > >
> > > The client has configured a response timeout of 1.5 seconds. In a
> > > case when our application tries to respond over a http2 stream
> > > asynchronously after 2 seconds where the client has already sent
> > > RST_STREAM with NO ERROR in 1.5 seconds
> >
> >
> > Why does the client send NO_ERROR to the server ? I think it should
> > send a CANCEL instead.
> > https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for
> > "Graceful shutdown of the server".
> >
> >
> > > (due to no response) , then tomcat sends GOAWAY and closes the HTTP2
> > > connection . Is this behavior of GOAWAY and connection closure
> expected ?
> > > We have planned to upgrade to Embedded tomcat version 9.0.38 . These
> > > are the behaviors we see in production with version 9.0.22 ,  so we
> > > need some help with analyzing / validating  the existing behavior
> > > before
> > the upgrade .
> > > Please let us know.
> > >
> >
> > Friendly advice:
> > Please setup 9.0.38 locally and test on it.
> > 9.0.22 is way too old. It is up to you to use it for your production
> > but for reporting bugs it is recommended to use the latest available
> version.
> > I, personally, prefer to spend my spare time with my family and
> > friends than to debug old versions just because the user doesn't
> > bother to test on a newer version.
> >
> >
> > >
> > > Thanks and Regards
> > > Arshiya Shariff
> > >
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Martin Grigorov]

On Thu, Sep 24, 2020 at 2:11 PM Martin Grigorov 
wrote:

> Hi,
>
> On Thu, Sep 24, 2020 at 1:02 PM Nils Breunese  wrote:
>
>> Hello,
>>
>> I recently learned that when a server that supports path parameters [0] —
>> like Tomcat (I found Jetty also does) — is run behind a reverse proxy that
>> does path-based access control checks and does not support path parameters,
>> your combined setup could be vulnerable.
>>
>> Consider this setup:
>>
>> 1. A Tomcat application without access restrictions
>> 2. An reverse proxy that only allows requests to /v1/* on the Tomcat
>> application. I’ve used Envoy on Kubernetes, configured via Istio’s
>> Role-Based Access Control (RBAC), but also observed this issue with other
>> proxies.
>>
>> Now I send a request for /v1/..;color=red/internal/secret to the proxy.
>>
>> - Envoy allows the request based on the /v1/* rule, because it does not
>> support path parameters, because they are not part of any recent standard
>> (RFC 2396 dropped them in 1998 [1])
>> - Tomcat parses ;color=red as a path parameter, resolves the rest of the
>> path to /v1/../internal/secret and consequently serves /internal/secret
>> - I’m pretty sure that whoever configured that RBAC policy did not want
>> this to be possible
>>
>> Note that:
>> - The client can request any URL on the Tomcat server via this path
>> traversal attack.
>> - This also works with ‘empty’ path parameters, e.g.
>> /v1/..;/internal/secret
>> - The URL doesn’t necessarily need to contain ‘..’ for naughtiness to be
>> possible. When access to /public/secret would be restricted by the proxy,
>> but other /public/* paths would be allowed, then an attacker would still be
>> able to get the contents of /public/secret by requesting /public;/secret
>>
>> The fact that different servers handle paths with path parameters
>> differently seems to be known [2], but especially setups in which reverse
>> proxies don’t support path parameters, but the servers behind them do,
>> seems risky.
>>
>> Would it be reasonable to create a ticket with a feature request for a
>> flag to disable path parameter support in Tomcat? Or an even more secure
>> suggestion: make path parameter support an opt-in feature, because it was
>> dropped from the URI standard in 1998 and can make setups like I described
>> vulnerable?
>>
>
> I remember a similar discussion from a few months ago.
> Which version of Tomcat do you use ? Please try with the latest in case
> you use an older one.
>

I've just checked my mail archives.
Someone else had the same/similar problem and the conclusion was that
according to the Servlet specification this is the proper way to process
the request - the request url should be normalized. If you need to protect
some paths then you should do whatever is necessary in your application.
Please use secur...@tomcat.apache.org for reporting (possible) security
problems in the future! Thanks!

Martin


>
>
>>
>> Thanks, Nils.
>>
>> [0] RFC 1738 https://tools.ietf.org/html/rfc1738 <
>> https://tools.ietf.org/html/rfc1738> (1994) and RFC 1808
>> https://tools.ietf.org/html/rfc1808 <https://tools.ietf.org/html/rfc1808>
>> (1995).
>> [1] RFC 2396 G.4. Modifications from RFC 1808
>> https://tools.ietf.org/html/rfc2396 <https://tools.ietf.org/html/rfc2396>
>> (1998). Path parameters are also absent in RFC 3986
>> https://tools.ietf.org/html/rfc3986 <https://tools.ietf.org/html/rfc3986>
>> (2005).
>> [2]
>> https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf
>> <
>> https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf>
>> slides 41 and further
>
>

Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Martin Grigorov]

Hi,

On Thu, Sep 24, 2020 at 1:02 PM Nils Breunese  wrote:

> Hello,
>
> I recently learned that when a server that supports path parameters [0] —
> like Tomcat (I found Jetty also does) — is run behind a reverse proxy that
> does path-based access control checks and does not support path parameters,
> your combined setup could be vulnerable.
>
> Consider this setup:
>
> 1. A Tomcat application without access restrictions
> 2. An reverse proxy that only allows requests to /v1/* on the Tomcat
> application. I’ve used Envoy on Kubernetes, configured via Istio’s
> Role-Based Access Control (RBAC), but also observed this issue with other
> proxies.
>
> Now I send a request for /v1/..;color=red/internal/secret to the proxy.
>
> - Envoy allows the request based on the /v1/* rule, because it does not
> support path parameters, because they are not part of any recent standard
> (RFC 2396 dropped them in 1998 [1])
> - Tomcat parses ;color=red as a path parameter, resolves the rest of the
> path to /v1/../internal/secret and consequently serves /internal/secret
> - I’m pretty sure that whoever configured that RBAC policy did not want
> this to be possible
>
> Note that:
> - The client can request any URL on the Tomcat server via this path
> traversal attack.
> - This also works with ‘empty’ path parameters, e.g.
> /v1/..;/internal/secret
> - The URL doesn’t necessarily need to contain ‘..’ for naughtiness to be
> possible. When access to /public/secret would be restricted by the proxy,
> but other /public/* paths would be allowed, then an attacker would still be
> able to get the contents of /public/secret by requesting /public;/secret
>
> The fact that different servers handle paths with path parameters
> differently seems to be known [2], but especially setups in which reverse
> proxies don’t support path parameters, but the servers behind them do,
> seems risky.
>
> Would it be reasonable to create a ticket with a feature request for a
> flag to disable path parameter support in Tomcat? Or an even more secure
> suggestion: make path parameter support an opt-in feature, because it was
> dropped from the URI standard in 1998 and can make setups like I described
> vulnerable?
>

I remember a similar discussion from a few months ago.
Which version of Tomcat do you use ? Please try with the latest in case you
use an older one.


>
> Thanks, Nils.
>
> [0] RFC 1738 https://tools.ietf.org/html/rfc1738 <
> https://tools.ietf.org/html/rfc1738> (1994) and RFC 1808
> https://tools.ietf.org/html/rfc1808 
> (1995).
> [1] RFC 2396 G.4. Modifications from RFC 1808
> https://tools.ietf.org/html/rfc2396 
> (1998). Path parameters are also absent in RFC 3986
> https://tools.ietf.org/html/rfc3986 
> (2005).
> [2]
> https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf
> <
> https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf>
> slides 41 and further

Re: Issue in unpacking files

[Martin Grigorov]

The response was inline:

The mailing list server does not allow embedded images.
Please use some image pastebin service.
Or if this is plain text then better just copy/paste it.

On Wed, Sep 23, 2020 at 3:35 PM Mohan T  wrote:

> Hi,
>
> I am not able to see any response in the mail received.
>
> Please treat the file name deployed as ems.war and not ems.wzr.
>
> Thanks
>
> Mohan
>
>
> -Original Message-
> From: Martin Grigorov 
> Sent: Wednesday, September 23, 2020 6:00 PM
> To: Tomcat Users List 
> Subject: Re: Issue in unpacking files
>
> Warning : This is an External Email. Check reliability of sender &  email
> id mentioned in ;
> use caution with links and attachments – IMG Team
>
>
>
>
>
>
> Hi,
>
> On Wed, Sep 23, 2020 at 3:03 PM Mohan T  wrote:
>
> > Dear All,
> >
> >
> >
> > We have two environments where in one the .war file is getting
> > extracted and on the other it is not. See the below for your
> > reference. The file being deployed is ems.wzr
> >
> >
> >
> > We are using apache-tomcat-8.5.35.  Any pointers would be of great help.
> >
> >
> >
> > Catalinabase0  - Here it has been correctly deployed -
> >
> >
> >
> >
> >
> > Catalinabase1 – Here it has not been deployed correctly -
> >
> >
> >
> >
>
> The mailing list server does not allow embedded images.
> Please use some image pastebin service.
> Or if this is plain text then better just copy/paste it.
>
>
> >
> >
> >
> > DISCLAIMER: This communication contains information which is
> > confidential and the copyright of Ramco Systems Ltd, its subsidiaries
> > or a third party (“Ramco”). This email may also contain legally
> privileged information.
> > Confidentiality and legal privilege attached to this communication are
> > not waived or lost by reason of mistaken delivery to you.This email is
> > intended to be read or used by the addressee only. If you are not the
> > intended recipient, any use, distribution, disclosure or copying of
> > this email is strictly prohibited without the express written approval
> > of Ramco. Please delete and destroy all copies and email Ramco at
> > le...@ramco.com immediately. Any views expressed in this communication
> > are those of the individual sender, except where the sender
> > specifically states them to be the views of Ramco. Except as required
> > by law, Ramco does not represent, warrant and/or guarantee that the
> > integrity of this communication has been maintained nor that the
> > communication is free of errors, virus, interception or interference.
> > If you do not wish to receive such communications, please forward this
> > communication to market...@ramco.com and express your wish not to
> receive such communications henceforth.
> >
> DISCLAIMER: This communication contains information which is confidential
> and the copyright of Ramco Systems Ltd, its subsidiaries or a third party
> (“Ramco”). This email may also contain legally privileged information.
> Confidentiality and legal privilege attached to this communication are not
> waived or lost by reason of mistaken delivery to you.This email is intended
> to be read or used by the addressee only. If you are not the intended
> recipient, any use, distribution, disclosure or copying of this email is
> strictly prohibited without the express written approval of Ramco. Please
> delete and destroy all copies and email Ramco at le...@ramco.com
> immediately. Any views expressed in this communication are those of the
> individual sender, except where the sender specifically states them to be
> the views of Ramco. Except as required by law, Ramco does not represent,
> warrant and/or guarantee that the integrity of this communication has been
> maintained nor that the communication is free of errors, virus,
> interception or interference. If you do not wish to receive such
> communications, please forward this communication to market...@ramco.com
> and express your wish not to receive such communications henceforth.
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: Issue in unpacking files

[Martin Grigorov]

Hi,

On Wed, Sep 23, 2020 at 3:03 PM Mohan T  wrote:

> Dear All,
>
>
>
> We have two environments where in one the .war file is getting extracted
> and on the other it is not. See the below for your reference. The file
> being deployed is ems.wzr
>
>
>
> We are using apache-tomcat-8.5.35.  Any pointers would be of great help.
>
>
>
> Catalinabase0  - Here it has been correctly deployed -
>
>
>
>
>
> Catalinabase1 – Here it has not been deployed correctly -
>
>
>
>

The mailing list server does not allow embedded images.
Please use some image pastebin service.
Or if this is plain text then better just copy/paste it.


>
>
>
> DISCLAIMER: This communication contains information which is confidential
> and the copyright of Ramco Systems Ltd, its subsidiaries or a third party
> (“Ramco”). This email may also contain legally privileged information.
> Confidentiality and legal privilege attached to this communication are not
> waived or lost by reason of mistaken delivery to you.This email is intended
> to be read or used by the addressee only. If you are not the intended
> recipient, any use, distribution, disclosure or copying of this email is
> strictly prohibited without the express written approval of Ramco. Please
> delete and destroy all copies and email Ramco at le...@ramco.com
> immediately. Any views expressed in this communication are those of the
> individual sender, except where the sender specifically states them to be
> the views of Ramco. Except as required by law, Ramco does not represent,
> warrant and/or guarantee that the integrity of this communication has been
> maintained nor that the communication is free of errors, virus,
> interception or interference. If you do not wish to receive such
> communications, please forward this communication to market...@ramco.com
> and express your wish not to receive such communications henceforth.
>

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi,



I've debugged further with Wireshark:

1) Golang: https://pasteboard.co/Jsnvx1z.png  (SETTINGS, HEADERS+DATA)
2) Nodejs: https://pasteboard.co/JsnwRNV.png (SETTINGS, HEADERS+PARTIAL
DATA, DATA)
3) Rust: https://pasteboard.co/JsnxvKGU.png  (SETTINGS, HEADERS+PARTIAL
DATA+DATA)
4) Netty: https://pasteboard.co/JsnxYJp.png (SETTINGS+HEADERS+DATA)
5) Tomcat: https://pasteboard.co/JsnyjpA.png (SETTINGS, HEADERS, DATA)

>From the images above you can see that only Tomcat sends two separate
packets for HEADERS and DATA frames. All others put several frames into one
packet.
Netty puts the SETTINGS, the HEADERS and the DATA into one TCP packet. I
guess this is the reason for the high throughput
Node.js writes the HEADERS and PARTIAL DATA in one packet and the actual
DATA in a separate one
Rust also uses PARTIAL DATA but puts the actual one in the same TCP packet
Tomcat's way is spec compliant but most probably is also a reason for the
lower throughput.

More findings:
I've debugged Netty code and it never receives RST_STREAM with CANCEL error
at
https://github.com/martin-g/http2-server-perf-tests/blob/2f628c407e2ea29be99e756a61e555d88addaa01/java/netty-http2-test/src/main/java/com/example/netty/http2/Http2ServerResponseHandler.java#L30
I've looked at Golang net code and found that it will send RST_STREAM with
CANCEL if there is a problem reading the body:
https://github.com/tsenart/vegeta/issues/540#issuecomment-696757695
It is not clear to me whether this is about the request body or response
body. But since the unit test uses Transport.RoundTrip() I think it is
about the response body.
So, it looks like Tomcat sends a DATA frame which is not liked by Golang
net package code and Golang sends this CANCEL error that causes the
problems.
The DATA frames in Wireshark look good to me, but I've made them by using
Firefox as a client to be able to decrypt TLS in Wireshark (SSLKEYLOGFILE)

Martin

Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR

[Martin Grigorov]

Hi,

On Tue, Sep 22, 2020 at 1:47 PM Arshiya Shariff
 wrote:

> Thank you so much Martin for the response.
>  Yes , 9.0.38 testing is on going .
>
> As we don’t get this clear with the RFC , please help us with the below
> two cases :

  * If a client sends RST_STREAM with NO_ERROR, then while sending async
> response is it expected behavior to close connection with GOAWAY ?
>   * If a client sends RST_STREAM with CANCEL , then while sending async
> response will tomcat send RST_STREAM or GOAWAY , from http2 protocol
> perspective ?
>

As per https://tools.ietf.org/html/rfc7540#section-6.4 and
https://tools.ietf.org/html/rfc7540#section-5.1 if "send R" or "recv R" are
processed then the stream is moved to CLOSED state.
GOAWAY should be sent to the client only if the connection will be closed (
https://tools.ietf.org/html/rfc7540#section-6.8). The server should close
the connection only if some serious problem has happened, e.g. an
IOException.

Tell us more about your use case. What do you want to do when 1.5secs pass
? What do you expect to happen ?


>
> Thanks and Regards
> Arshiya Sharif
>
> -Original Message-
> From: Martin Grigorov 
> Sent: Tuesday, September 22, 2020 1:18 AM
> To: Tomcat Users List 
> Subject: Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a
> stream where the client has already sent RST_STREAM:NO ERROR
>
> Hi,
>
> On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff <
> arshiya.shar...@ericsson.com.invalid> wrote:
>
> > Hi All,
> >
> > The client has configured a response timeout of 1.5 seconds. In a case
> > when our application tries to respond over a http2 stream
> > asynchronously after 2 seconds where the client has already sent
> > RST_STREAM with NO ERROR in 1.5 seconds
>
>
> Why does the client send NO_ERROR to the server ? I think it should send a
> CANCEL instead.
> https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for "Graceful
> shutdown of the server".
>
>
> > (due to no response) , then tomcat sends GOAWAY and closes the HTTP2
> > connection . Is this behavior of GOAWAY and connection closure expected ?
> > We have planned to upgrade to Embedded tomcat version 9.0.38 . These
> > are the behaviors we see in production with version 9.0.22 ,  so we
> > need some help with analyzing / validating  the existing behavior before
> the upgrade .
> > Please let us know.
> >
>
> Friendly advice:
> Please setup 9.0.38 locally and test on it.
> 9.0.22 is way too old. It is up to you to use it for your production but
> for reporting bugs it is recommended to use the latest available version.
> I, personally, prefer to spend my spare time with my family and friends
> than to debug old versions just because the user doesn't bother to test on
> a newer version.
>
>
> >
> > Thanks and Regards
> > Arshiya Shariff
> >
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 4:46 PM Rémy Maucherat  wrote:

> On Mon, Sep 21, 2020 at 2:49 PM Martin Grigorov 
> wrote:
>
> > Hi Remy,
> >
> > On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
> >
> > 
> >
> >
> > > > 2020-09-21 14:25:04.850 DEBUG 232086 ---
> [https-jsse-nio-18080-exec-8]
> > > > o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
> > > > socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
> > > > :java.nio.channels.SocketChannel[connected
> > > > local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
> > > > 2020-09-21 14:25:04.850 DEBUG 232086 ---
> [https-jsse-nio-18080-exec-6]
> > > > o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
> > > >
> > > > java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
> > > > at org.apache.tomcat.util.net
> > > > .SecureNioChannel.read(SecureNioChannel.java:766)
> > > > at org.apache.tomcat.util.net
> > > >
> > >
> >
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> > > >
> > >
> > > When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
> > > what's going on. Otherwise you don't really know what proportion of TLS
> > or
> > > HTTP/2 impacts performance more [and then you have to test more things
> > like
> > > OpenSSL, be careful that the ciphers used are the same and equally well
> > > optimized in each impl, etc].
> > >
> > > Then once you feel happy about h2c, you can see how things go with TLS.
> > >
> >
> > Chris also suggested that TLS might be the problem for the low throughput
> > but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
> > reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear
> now
> > - it is due to the extra RST packets being sent.
> >
> > Vegeta does support h2c! So I can give it a try!
> >
> > How one configures Tomcat to support h2c ? Just remove  ?
> > I've just tried this but I am not sure how to verify that it works.
> > The browsers do not support h2c. curl and httpie neither.
> > Quick search suggested me https://github.com/fstab/h2c "h2c connect
> > localhost:18080" failed with "Failed to connect to localhost:18080: tls:
> > oversized record received with length 20527"
> >
>
> Yes, it depends on the client. h2load works just fine without TLS for
> example, and it's actually the reason why I added h2c support in Tomcat.
>
> As for the configuration, it's a normal non TLS HTTP/1.1 config with the
> UpgradeProtocol element added.
>

The way to use h2c with curl is: curl -i --http2-prior-knowledge
http://localhost:18080/...

(with my change in reset() to avoid the connection closes):

vegeta ... -http2=f -h2c ... gives 13 K reqs/s
vegeta ... -http2 ... gives 8.2 K reqs/s

so, it seems TLS reduces the throughput with 40%

Martin


>
> Rémy
>
>
> >
> >
> >
> > >
> > > Rémy
> > >
> > >
> > >
> > >
> >
>

Re: HTTP2: Tomcat sends GOAWAY when trying to respond over a stream where the client has already sent RST_STREAM:NO ERROR

[Martin Grigorov]

Hi,

On Mon, Sep 21, 2020 at 7:56 PM Arshiya Shariff
 wrote:

> Hi All,
>
> The client has configured a response timeout of 1.5 seconds. In a case
> when our application tries to respond over a http2 stream asynchronously
> after 2 seconds where the client has already sent RST_STREAM with NO ERROR
> in 1.5 seconds


Why does the client send NO_ERROR to the server ? I think it should send a
CANCEL instead.
https://tools.ietf.org/html/rfc7540 mentions NO_ERROR only for "Graceful
shutdown of the server".


> (due to no response) , then tomcat sends GOAWAY and closes the HTTP2
> connection . Is this behavior of GOAWAY and connection closure expected ?
> We have planned to upgrade to Embedded tomcat version 9.0.38 . These are
> the behaviors we see in production with version 9.0.22 ,  so we need some
> help with analyzing / validating  the existing behavior before the upgrade .
> Please let us know.
>

Friendly advice:
Please setup 9.0.38 locally and test on it.
9.0.22 is way too old. It is up to you to use it for your production but
for reporting bugs it is recommended to use the latest available version.
I, personally, prefer to spend my spare time with my family and friends
than to debug old versions just because the user doesn't bother to test on
a newer version.


>
> Thanks and Regards
> Arshiya Shariff
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 9:31 PM Martin Grigorov 
wrote:

>
>
> On Mon, Sep 21, 2020 at 5:52 PM Mark Thomas  wrote:
>
>> On 21/09/2020 13:48, Martin Grigorov wrote:
>> > Hi Remy,
>> >
>> > On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
>> >
>> > 
>> >
>> >
>> >>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
>> >>> o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
>> >>> socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
>> >>> :java.nio.channels.SocketChannel[connected
>> >>> local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
>> >>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
>> >>> o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
>> >>>
>> >>> java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
>> >>> at org.apache.tomcat.util.net
>> >>> .SecureNioChannel.read(SecureNioChannel.java:766)
>> >>> at org.apache.tomcat.util.net
>> >>>
>> >>
>> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
>> >>>
>> >>
>> >> When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
>> >> what's going on. Otherwise you don't really know what proportion of
>> TLS or
>> >> HTTP/2 impacts performance more [and then you have to test more things
>> like
>> >> OpenSSL, be careful that the ciphers used are the same and equally well
>> >> optimized in each impl, etc].
>> >>
>> >> Then once you feel happy about h2c, you can see how things go with TLS.
>> >>
>> >
>> > Chris also suggested that TLS might be the problem for the low
>> throughput
>> > but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
>> > reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear
>> now
>> > - it is due to the extra RST packets being sent.
>> >
>> > Vegeta does support h2c! So I can give it a try!
>>
>> I've just done that. I am seeing the client sending RST_STREAM frames
>> AFTER the request/response has been processed cleanly.
>>
>> The RST_STREAM frames are being sent with the CANCEL error. i.e. the
>> client is no longer interested in reading the response body.
>>
>> This doesn't make much sense as there is little more than two or three
>> milliseconds between the start of the request and the RST_STREAM. It
>> isn't happening due to a timeout. It is also always sent AFTER the
>> server has sent the complete response (the end of stream flag is set)
>> and sometimes after the next request has been sent.
>>
>> However you look at it, this looks like a bug in Vegeta. That doesn't
>>
>
> I will contact the Vegeta community!
>

https://github.com/tsenart/vegeta/issues/540


>
>
>> exclude, of course, the possibility of improving how Tomcat handles
>> traffic like this.
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 5:52 PM Mark Thomas  wrote:

> On 21/09/2020 13:48, Martin Grigorov wrote:
> > Hi Remy,
> >
> > On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
> >
> > 
> >
> >
> >>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
> >>> o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
> >>> socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
> >>> :java.nio.channels.SocketChannel[connected
> >>> local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
> >>> 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
> >>> o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
> >>>
> >>> java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
> >>> at org.apache.tomcat.util.net
> >>> .SecureNioChannel.read(SecureNioChannel.java:766)
> >>> at org.apache.tomcat.util.net
> >>>
> >>
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> >>>
> >>
> >> When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
> >> what's going on. Otherwise you don't really know what proportion of TLS
> or
> >> HTTP/2 impacts performance more [and then you have to test more things
> like
> >> OpenSSL, be careful that the ciphers used are the same and equally well
> >> optimized in each impl, etc].
> >>
> >> Then once you feel happy about h2c, you can see how things go with TLS.
> >>
> >
> > Chris also suggested that TLS might be the problem for the low throughput
> > but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
> > reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear
> now
> > - it is due to the extra RST packets being sent.
> >
> > Vegeta does support h2c! So I can give it a try!
>
> I've just done that. I am seeing the client sending RST_STREAM frames
> AFTER the request/response has been processed cleanly.
>
> The RST_STREAM frames are being sent with the CANCEL error. i.e. the
> client is no longer interested in reading the response body.
>
> This doesn't make much sense as there is little more than two or three
> milliseconds between the start of the request and the RST_STREAM. It
> isn't happening due to a timeout. It is also always sent AFTER the
> server has sent the complete response (the end of stream flag is set)
> and sometimes after the next request has been sent.
>
> However you look at it, this looks like a bug in Vegeta. That doesn't
>

I will contact the Vegeta community!


> exclude, of course, the possibility of improving how Tomcat handles
> traffic like this.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 3:48 PM Martin Grigorov 
wrote:

> Hi Remy,
>
> On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:
>
> 
>
>
>> > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
>> > o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
>> > socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
>> > :java.nio.channels.SocketChannel[connected
>> > local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
>> > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
>> > o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
>> >
>> > java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
>> > at org.apache.tomcat.util.net
>> > .SecureNioChannel.read(SecureNioChannel.java:766)
>> > at org.apache.tomcat.util.net
>> >
>> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
>> >
>>
>> When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
>> what's going on. Otherwise you don't really know what proportion of TLS or
>> HTTP/2 impacts performance more [and then you have to test more things
>> like
>> OpenSSL, be careful that the ciphers used are the same and equally well
>> optimized in each impl, etc].
>>
>> Then once you feel happy about h2c, you can see how things go with TLS.
>>
>
> Chris also suggested that TLS might be the problem for the low throughput
> but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
> reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear now
> - it is due to the extra RST packets being sent.
>
> Vegeta does support h2c! So I can give it a try!
>
> How one configures Tomcat to support h2c ? Just remove  ?
> I've just tried this but I am not sure how to verify that it works.
> The browsers do not support h2c. curl and httpie neither.
> Quick search suggested me https://github.com/fstab/h2c "h2c connect
> localhost:18080" failed with "Failed to connect to localhost:18080: tls:
> oversized record received with length 20527"
>

fstab/h2c is a normal http2 client. Confusingly named one though.


>
>
>>
>> Rémy
>>
>>
>>
>>

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi Remy,

On Mon, Sep 21, 2020 at 2:56 PM Rémy Maucherat  wrote:




> > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-8]
> > o.a.coyote.http11.Http11NioProtocol  : Found processor [null] for
> > socket [org.apache.tomcat.util.net.SecureNioChannel@2b435926
> > :java.nio.channels.SocketChannel[connected
> > local=/127.0.0.1:18080 remote=/127.0.0.1:42229]]
> > 2020-09-21 14:25:04.850 DEBUG 232086 --- [https-jsse-nio-18080-exec-6]
> > o.a.coyote.http2.Http2UpgradeHandler : Connection [2]
> >
> > java.io.IOException: Unable to unwrap data, invalid status [CLOSED]
> > at org.apache.tomcat.util.net
> > .SecureNioChannel.read(SecureNioChannel.java:766)
> > at org.apache.tomcat.util.net
> >
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
> >
>
> When I tested HTTP/2, I used h2c (unencrypted) to get a better idea at
> what's going on. Otherwise you don't really know what proportion of TLS or
> HTTP/2 impacts performance more [and then you have to test more things like
> OpenSSL, be careful that the ciphers used are the same and equally well
> optimized in each impl, etc].
>
> Then once you feel happy about h2c, you can see how things go with TLS.
>

Chris also suggested that TLS might be the problem for the low throughput
but I get good throughput for both HTTP (15-16 K) and HTTPS (12-13 K
reqs/s). Only when HTTP2 is enabled it drops. The reason is more clear now
- it is due to the extra RST packets being sent.

Vegeta does support h2c! So I can give it a try!

How one configures Tomcat to support h2c ? Just remove  ?
I've just tried this but I am not sure how to verify that it works.
The browsers do not support h2c. curl and httpie neither.
Quick search suggested me https://github.com/fstab/h2c "h2c connect
localhost:18080" failed with "Failed to connect to localhost:18080: tls:
oversized record received with length 20527"



>
> Rémy
>
>
>
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 12:08 PM Martin Grigorov 
wrote:

>
>
> On Mon, Sep 21, 2020 at 11:23 AM Mark Thomas  wrote:
>
>> On 21/09/2020 08:18, Martin Grigorov wrote:
>> > On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:
>> >
>> >> On 18/09/2020 14:07, Martin Grigorov wrote:
>> >>
>> >> 
>> >>
>> >>> What is the difference
>> >>> between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
>> >>> and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
>> >>
>> >> Compare the parameters used to construct the enums.
>> >>
>> >>> I read some parts of https://tools.ietf.org/html/rfc7540 but I don't
>> see
>> >>> anything related to two types of CLOSED state.
>> >>
>> >> Section 5.1. Definition of the closed state (page 20) explains the
>> >> difference between the two.
>> >>
>> >
>> > Still I do not understand what RX and TX stand for. But this is not
>> > important.
>>
>> TX - transmit
>>
>> RX - receive
>>
>
> Thanks!
>
>
>>
>>
>> > The following patch fixes the issue for me/Vegeta:
>> >
>> > @@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends AbstractStream
>> > implements InternalHttpUpgradeH
>> >
>> >  @Override
>> >  public void reset(int streamId, long errorCode) throws
>> Http2Exception
>> >  {
>> > -Stream stream = getStream(streamId, true);
>> > -boolean active = stream.isActive();
>> > -stream.checkState(FrameType.RST);
>> > -stream.receiveReset(errorCode);
>> > -if (active) {
>> > -activeRemoteStreamCount.decrementAndGet();
>> > +boolean unknownIsError = Http2Error.CANCEL.getCode() !=
>> errorCode;
>> > +Stream stream = getStream(streamId, unknownIsError);
>> > +if (stream != null) {
>> > +boolean active = stream.isActive();
>> > +stream.checkState(FrameType.RST);
>> > +stream.receiveReset(errorCode);
>> > +if (active) {
>> > +activeRemoteStreamCount.decrementAndGet();
>> > +}
>> >  }
>> >  }
>> >
>> > I.e. do not throw an error if the remote peer is trying to cancel an
>> > already closed stream.
>>
>> RFC 7540 allows the connection to be closed with a protocol error if the
>> reset is received "a significant time after sending END_STREAM".
>>
>> Tomcat retains state for maxConcurrentStreams + 10% with closed streams
>> being pruned (oldest first) every 10 new streams.
>>
>
> The spec talks about time but Tomcat uses the number of streams.
> I understand that under normal usage this could be enough "time" but under
> heavy load this number is reached quickly and the time is short.
>
> I've tried with these values for
> maxConcurrentStreams/maxConcurrentStreamExecution: 8 (number of CPU cores),
> 16, 100 (the default), 1024 and 8196. The throughput is around 10K only
> until 100. After that it drops (as you said earlier).
>
>
>>
>> It isn't clear at this point why the client is sending the RST_STREAM.
>>
>
> I don't know either. But Vegeta is using the Golang standard libraries, so
> any Golang application should behave the same way.
> Also as I investigated last week - Golang/Node.js/Rust/Netty HTTP2 servers
> do not complain about it.
>
>
>> This normally indicates an error condition. From the description of the
>> load test I don't see why the client would be sending a RST_STREAM.
>>
>> If the RST_STREAM is valid and it isn't received a significant time
>> after the END_STREAM then Tomcat needs to handle this.
>>
>> If the RST_STREAM is valid but received a significant amount of time
>> after the END_STREAM that would be a client error would be a client error.
>>
>> Of course, significant isn't defined in the specification.
>>
>> Any special handling for RST_STREAM also needs to be applied to
>> WINDOW_UPDATE. It also needs to differentiate between a frame received
>> for a past closed stream and a frame received for an stream that never
>> existed.
>>
>> I think the key question here is why is the client sending the
>> RST_STREAM in the first place. Is Tomcat doing something it shouldn't /
>> not doing something it should to cause this? If so, we need to fix the
>&

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 12:08 PM Martin Grigorov 
wrote:

>
>
> On Mon, Sep 21, 2020 at 11:23 AM Mark Thomas  wrote:
>
>> On 21/09/2020 08:18, Martin Grigorov wrote:
>> > On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:
>> >
>> >> On 18/09/2020 14:07, Martin Grigorov wrote:
>> >>
>> >> 
>> >>
>> >>> What is the difference
>> >>> between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
>> >>> and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
>> >>
>> >> Compare the parameters used to construct the enums.
>> >>
>> >>> I read some parts of https://tools.ietf.org/html/rfc7540 but I don't
>> see
>> >>> anything related to two types of CLOSED state.
>> >>
>> >> Section 5.1. Definition of the closed state (page 20) explains the
>> >> difference between the two.
>> >>
>> >
>> > Still I do not understand what RX and TX stand for. But this is not
>> > important.
>>
>> TX - transmit
>>
>> RX - receive
>>
>
> Thanks!
>
>
>>
>>
>> > The following patch fixes the issue for me/Vegeta:
>> >
>> > @@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends AbstractStream
>> > implements InternalHttpUpgradeH
>> >
>> >  @Override
>> >  public void reset(int streamId, long errorCode) throws
>> Http2Exception
>> >  {
>> > -Stream stream = getStream(streamId, true);
>> > -boolean active = stream.isActive();
>> > -stream.checkState(FrameType.RST);
>> > -stream.receiveReset(errorCode);
>> > -if (active) {
>> > -activeRemoteStreamCount.decrementAndGet();
>> > +boolean unknownIsError = Http2Error.CANCEL.getCode() !=
>> errorCode;
>> > +Stream stream = getStream(streamId, unknownIsError);
>> > +if (stream != null) {
>> > +boolean active = stream.isActive();
>> > +stream.checkState(FrameType.RST);
>> > +stream.receiveReset(errorCode);
>> > +if (active) {
>> > +activeRemoteStreamCount.decrementAndGet();
>> > +}
>> >  }
>> >  }
>> >
>> > I.e. do not throw an error if the remote peer is trying to cancel an
>> > already closed stream.
>>
>> RFC 7540 allows the connection to be closed with a protocol error if the
>> reset is received "a significant time after sending END_STREAM".
>>
>> Tomcat retains state for maxConcurrentStreams + 10% with closed streams
>> being pruned (oldest first) every 10 new streams.
>>
>
> The spec talks about time but Tomcat uses the number of streams.
> I understand that under normal usage this could be enough "time" but under
> heavy load this number is reached quickly and the time is short.
>
> I've tried with these values for
> maxConcurrentStreams/maxConcurrentStreamExecution: 8 (number of CPU cores),
> 16, 100 (the default), 1024 and 8196. The throughput is around 10K only
> until 100. After that it drops (as you said earlier).
>
>
>>
>> It isn't clear at this point why the client is sending the RST_STREAM.
>>
>
> I don't know either. But Vegeta is using the Golang standard libraries, so
> any Golang application should behave the same way.
> Also as I investigated last week - Golang/Node.js/Rust/Netty HTTP2 servers
> do not complain about it.
>
>
>> This normally indicates an error condition. From the description of the
>> load test I don't see why the client would be sending a RST_STREAM.
>>
>> If the RST_STREAM is valid and it isn't received a significant time
>> after the END_STREAM then Tomcat needs to handle this.
>>
>> If the RST_STREAM is valid but received a significant amount of time
>> after the END_STREAM that would be a client error would be a client error.
>>
>> Of course, significant isn't defined in the specification.
>>
>> Any special handling for RST_STREAM also needs to be applied to
>> WINDOW_UPDATE. It also needs to differentiate between a frame received
>> for a past closed stream and a frame received for an stream that never
>> existed.
>>
>> I think the key question here is why is the client sending the
>> RST_STREAM in the first place. Is Tomcat doing something it shouldn't /
>> not doing something it should to cause this? If so, we need to fix the
>&g

Re: Low throughput with HTTP2

[Martin Grigorov]

On Mon, Sep 21, 2020 at 11:23 AM Mark Thomas  wrote:

> On 21/09/2020 08:18, Martin Grigorov wrote:
> > On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:
> >
> >> On 18/09/2020 14:07, Martin Grigorov wrote:
> >>
> >> 
> >>
> >>> What is the difference
> >>> between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
> >>> and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
> >>
> >> Compare the parameters used to construct the enums.
> >>
> >>> I read some parts of https://tools.ietf.org/html/rfc7540 but I don't
> see
> >>> anything related to two types of CLOSED state.
> >>
> >> Section 5.1. Definition of the closed state (page 20) explains the
> >> difference between the two.
> >>
> >
> > Still I do not understand what RX and TX stand for. But this is not
> > important.
>
> TX - transmit
>
> RX - receive
>

Thanks!


>
>
> > The following patch fixes the issue for me/Vegeta:
> >
> > @@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends AbstractStream
> > implements InternalHttpUpgradeH
> >
> >  @Override
> >  public void reset(int streamId, long errorCode) throws
> Http2Exception
> >  {
> > -Stream stream = getStream(streamId, true);
> > -boolean active = stream.isActive();
> > -stream.checkState(FrameType.RST);
> > -stream.receiveReset(errorCode);
> > -if (active) {
> > -activeRemoteStreamCount.decrementAndGet();
> > +boolean unknownIsError = Http2Error.CANCEL.getCode() !=
> errorCode;
> > +Stream stream = getStream(streamId, unknownIsError);
> > +if (stream != null) {
> > +boolean active = stream.isActive();
> > +stream.checkState(FrameType.RST);
> > +stream.receiveReset(errorCode);
> > +if (active) {
> > +activeRemoteStreamCount.decrementAndGet();
> > +}
> >  }
> >  }
> >
> > I.e. do not throw an error if the remote peer is trying to cancel an
> > already closed stream.
>
> RFC 7540 allows the connection to be closed with a protocol error if the
> reset is received "a significant time after sending END_STREAM".
>
> Tomcat retains state for maxConcurrentStreams + 10% with closed streams
> being pruned (oldest first) every 10 new streams.
>

The spec talks about time but Tomcat uses the number of streams.
I understand that under normal usage this could be enough "time" but under
heavy load this number is reached quickly and the time is short.

I've tried with these values for
maxConcurrentStreams/maxConcurrentStreamExecution: 8 (number of CPU cores),
16, 100 (the default), 1024 and 8196. The throughput is around 10K only
until 100. After that it drops (as you said earlier).


>
> It isn't clear at this point why the client is sending the RST_STREAM.
>

I don't know either. But Vegeta is using the Golang standard libraries, so
any Golang application should behave the same way.
Also as I investigated last week - Golang/Node.js/Rust/Netty HTTP2 servers
do not complain about it.


> This normally indicates an error condition. From the description of the
> load test I don't see why the client would be sending a RST_STREAM.
>
> If the RST_STREAM is valid and it isn't received a significant time
> after the END_STREAM then Tomcat needs to handle this.
>
> If the RST_STREAM is valid but received a significant amount of time
> after the END_STREAM that would be a client error would be a client error.
>
> Of course, significant isn't defined in the specification.
>
> Any special handling for RST_STREAM also needs to be applied to
> WINDOW_UPDATE. It also needs to differentiate between a frame received
> for a past closed stream and a frame received for an stream that never
> existed.
>
> I think the key question here is why is the client sending the
> RST_STREAM in the first place. Is Tomcat doing something it shouldn't /
> not doing something it should to cause this? If so, we need to fix the
> root cause rather than tackle the symptom.
>
> There are a couple of options for debugging this to see why the client
> is sending the RST_STREAM:
> - Enable debug logging for HTTP/2 in Tomcat. This is very verbose and
>   if the root cause is threading related typically changes the timing
>   enough to mask the issue.
>

Here are the logs of 20 seconds load:
https://gist.githubusercontent.com/martin-g/37dacc12149d5cecbfd4e16dc2248cfd/raw/76693cd1dd1f37ba126edb84ebd7631802aa91a6/tomcat-http2.log

This is with my pa

Re: Low throughput with HTTP2

[Martin Grigorov]

On Fri, Sep 18, 2020 at 6:16 PM Mark Thomas  wrote:

> On 18/09/2020 14:07, Martin Grigorov wrote:
>
> 
>
> > What is the difference
> > between org.apache.coyote.http2.StreamStateMachine.State#CLOSED_RX
> > and org.apache.coyote.http2.StreamStateMachine.State#CLOSED_TX ?
>
> Compare the parameters used to construct the enums.
>
> > I read some parts of https://tools.ietf.org/html/rfc7540 but I don't see
> > anything related to two types of CLOSED state.
>
> Section 5.1. Definition of the closed state (page 20) explains the
> difference between the two.
>

Still I do not understand what RX and TX stand for. But this is not
important.

The following patch fixes the issue for me/Vegeta:

@@ -1570,12 +1571,15 @@ class Http2UpgradeHandler extends AbstractStream
implements InternalHttpUpgradeH

 @Override
 public void reset(int streamId, long errorCode) throws Http2Exception
 {
-Stream stream = getStream(streamId, true);
-boolean active = stream.isActive();
-stream.checkState(FrameType.RST);
-stream.receiveReset(errorCode);
-if (active) {
-activeRemoteStreamCount.decrementAndGet();
+boolean unknownIsError = Http2Error.CANCEL.getCode() != errorCode;
+Stream stream = getStream(streamId, unknownIsError);
+if (stream != null) {
+boolean active = stream.isActive();
+stream.checkState(FrameType.RST);
+stream.receiveReset(errorCode);
+if (active) {
+activeRemoteStreamCount.decrementAndGet();
+}
 }
 }

I.e. do not throw an error if the remote peer is trying to cancel an
already closed stream.

With this change and Vegeta's -max-workers=100 I can get 12 K reqs/sec.

With more workers it starts failing with:

"status_codes": {
"0": 1000,
"200": 1
  },
  "errors": [
"Get \"https://localhost:18080/testbed/plaintext\": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)",
"Get \"https://localhost:18080/testbed/plaintext\": context deadline
exceeded",
"Get \"https://localhost:18080/testbed/plaintext\": context deadline
exceeded (Client.Timeout exceeded while awaiting headers)"
  ]
i.e. out of 1001 requests only one succeeds and the others fail with
timeout. I will try to debug this one too and run a profiler too.



>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Low throughput with HTTP2

[Martin Grigorov]

On Fri, Sep 18, 2020 at 10:48 AM Martin Grigorov 
wrote:

> Hi,
>
> On Wed, Sep 16, 2020 at 10:24 AM Martin Grigorov 
> wrote:
>
>> Hi Remy,
>>
>> On Tue, Sep 15, 2020 at 6:16 PM Rémy Maucherat  wrote:
>>
>>> On Tue, Sep 15, 2020 at 5:08 PM Martin Grigorov 
>>> wrote:
>>>
>>> > Hi Mark,
>>> >
>>> > On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas  wrote:
>>> >
>>> > > On 15/09/2020 12:46, Martin Grigorov wrote:
>>> > > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov <
>>> mgrigo...@apache.org>
>>> > > > wrote:
>>> > > >
>>> > > >> Hi,
>>> > > >>
>>> > > >> I am running some load tests on Tomcat and I've noticed that when
>>> > HTTP2
>>> > > is
>>> > > >> enabled the throughput drops considerably.
>>> > > >>
>>> > > >> Here are the steps to reproduce:
>>> > > >>
>>> > > >> 1) Enable HTTP2, e.g. by commenting out this connector:
>>> > > >>
>>> > > >>
>>> > >
>>> >
>>> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
>>> > > >>
>>> > > >> 2) Download Vegeta load tool from:
>>> > > >> https://github.com/tsenart/vegeta/releases/
>>> > > >>
>>> > > >> 3) Run the load tests:
>>> > > >>
>>> > > >> 3.1) HTTP/1.1
>>> > > >> echo -e '{"method": "GET", "url": "
>>> http://localhost:8080/examples/"}'
>>> > |
>>> > > >> vegeta attack -format=json  -rate=0 -max-workers=1000
>>> -duration=10s |
>>> > > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json
>>> > > >> /tmp/http1.json | jq .
>>> > > >>
>>> > > >> 3.2) HTTP2
>>> > > >> echo -e '{"method": "GET", "url": "
>>> https://localhost:8443/examples/
>>> > "}'
>>> > > |
>>> > > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000
>>> -insecure
>>> > > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
>>> > > >> -type=json /tmp/http2.json | jq .
>>> > > >>
>>> > > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0
>>> means
>>> > > >> that Vegeta will try to send as many requests as possible with the
>>> > > >> configured number of workers.
>>> > > >> I use '-insecure' because I use self-signed certificate.
>>> > > >>
>>> > > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only
>>> > > responses
>>> > > >> with code=200 .
>>> > > >> But for HTTP2 Tomcat starts returning such kind of errors:
>>> > > >>
>>> > > >>  "errors": [
>>> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
>>> > > GOAWAY
>>> > > >> and closed the connection; LastStreamID=9259,
>>> ErrCode=PROTOCOL_ERROR,
>>> > > >> debug=\"Stream [9,151] has been closed for some time\"",
>>> > > >> "http2: server sent GOAWAY and closed the connection;
>>> > > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151]
>>> has
>>> > > been
>>> > > >> closed for some time\"",
>>> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
>>> > > GOAWAY
>>> > > >> and closed the connection; LastStreamID=239,
>>> ErrCode=PROTOCOL_ERROR,
>>> > > >> debug=\"Stream [49] has been closed for some time\""
>>> > > >>   ]
>>> > > >>
>>> > > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
>>> > >
>>> > > That indicates that the client has sent a frame associated with a
>>> stream
>>> > > that the server closed previously and that that stream has been
>>>

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi,

On Wed, Sep 16, 2020 at 10:24 AM Martin Grigorov 
wrote:

> Hi Remy,
>
> On Tue, Sep 15, 2020 at 6:16 PM Rémy Maucherat  wrote:
>
>> On Tue, Sep 15, 2020 at 5:08 PM Martin Grigorov 
>> wrote:
>>
>> > Hi Mark,
>> >
>> > On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas  wrote:
>> >
>> > > On 15/09/2020 12:46, Martin Grigorov wrote:
>> > > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov <
>> mgrigo...@apache.org>
>> > > > wrote:
>> > > >
>> > > >> Hi,
>> > > >>
>> > > >> I am running some load tests on Tomcat and I've noticed that when
>> > HTTP2
>> > > is
>> > > >> enabled the throughput drops considerably.
>> > > >>
>> > > >> Here are the steps to reproduce:
>> > > >>
>> > > >> 1) Enable HTTP2, e.g. by commenting out this connector:
>> > > >>
>> > > >>
>> > >
>> >
>> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
>> > > >>
>> > > >> 2) Download Vegeta load tool from:
>> > > >> https://github.com/tsenart/vegeta/releases/
>> > > >>
>> > > >> 3) Run the load tests:
>> > > >>
>> > > >> 3.1) HTTP/1.1
>> > > >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/
>> "}'
>> > |
>> > > >> vegeta attack -format=json  -rate=0 -max-workers=1000
>> -duration=10s |
>> > > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json
>> > > >> /tmp/http1.json | jq .
>> > > >>
>> > > >> 3.2) HTTP2
>> > > >> echo -e '{"method": "GET", "url": "
>> https://localhost:8443/examples/
>> > "}'
>> > > |
>> > > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000
>> -insecure
>> > > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
>> > > >> -type=json /tmp/http2.json | jq .
>> > > >>
>> > > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0
>> means
>> > > >> that Vegeta will try to send as many requests as possible with the
>> > > >> configured number of workers.
>> > > >> I use '-insecure' because I use self-signed certificate.
>> > > >>
>> > > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only
>> > > responses
>> > > >> with code=200 .
>> > > >> But for HTTP2 Tomcat starts returning such kind of errors:
>> > > >>
>> > > >>  "errors": [
>> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
>> > > GOAWAY
>> > > >> and closed the connection; LastStreamID=9259,
>> ErrCode=PROTOCOL_ERROR,
>> > > >> debug=\"Stream [9,151] has been closed for some time\"",
>> > > >> "http2: server sent GOAWAY and closed the connection;
>> > > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151]
>> has
>> > > been
>> > > >> closed for some time\"",
>> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
>> > > GOAWAY
>> > > >> and closed the connection; LastStreamID=239,
>> ErrCode=PROTOCOL_ERROR,
>> > > >> debug=\"Stream [49] has been closed for some time\""
>> > > >>   ]
>> > > >>
>> > > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
>> > >
>> > > That indicates that the client has sent a frame associated with a
>> stream
>> > > that the server closed previously and that that stream has been
>> removed
>> > > from the Map of known streams to make room for new ones. See
>> > > Http2UpgardeHandler.pruneClosedStreams()
>> > >
>> > > It looks like the client is making assumptions about server behaviour
>> > > that go beyond the requirements of RFC 7540, section 5.3.4.
>> > >
>> >
>> > This is pos

Re: tomcat warnings. [EXTERNAL]

[Martin Grigorov]

On Thu, Sep 17, 2020, 16:07 Beard, Shawn 
wrote:

> Yes its 9.0.31.0
>

The last 0 makes me think that you have installed Tomcat by using your OS
package manager.
Try with stock Tomcat downloaded from Apache servers.


> [mwuser@usilg01-tcd003 ~]$ ./version.sh
> Using CATALINA_BASE:   /path/to/catalina_base
> Using CATALINA_HOME:   /path/to/catalina_home
> Using CATALINA_TMPDIR: /path/to/catalina_base/temp
> Using JRE_HOME:/
> Using CLASSPATH:   /path/to/catalina_base
> /bin/bootstrap.jar:/usr/apache/tomcat/tomcat-current/bin/tomcat-juli.jar
> Server version: Apache Tomcat/9.0.31
> Server built:   Feb 5 2020 19:32:12 UTC
> Server number:  9.0.31.0
> OS Name:Linux
> OS Version: 3.10.0-957.21.3.el7.x86_64
> Architecture:   amd64
> JVM Version:1.8.0_212-b04
> JVM Vendor: Oracle Corporation
>
> I do have 3 connectors, one of them is the AJP connector, could the
> warning be coming from them?
>

Remove the problematic attributes from each connector one by one and see
when the warnings will disappear.


> Below is all 3 connectors:
>
> connectionTimeout="2"
>Server=" "
>maxHttpHeaderSize="8192"
>maxThreads="500"
>minSpareThreads="30"
>enableLookups="false"
>disableUploadTimeout="true"
>acceptCount="150"
>redirectPort="9443"
>
>  
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
>compression="on"
>compressionMinSize="2048"
>noCompressionUserAgents="gozilla,traviata" />
>
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>connectionTimeout="2"
>Server=" "
>maxHttpHeaderSize="8192"
>maxThreads="500"
>minSpareThreads="30"
>enableLookups="false"
>disableUploadTimeout="true"
>acceptCount="150"
>scheme="https"
>secure="true"
>SSLEnabled="true"
>sslEnabledProtocols="+TLSv1.2"
>keystoreFile="/path/to/tuststore/TomcatTrustStore.p12"
>truststoreFile="/path/to/keystore/TomcatTrustStore.p12"
>keystorePass="XXX"
>truststorePass="XX"
>
> ciphers="TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_RC4_128_SHA"
>
>  
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
>compression="on"
>compressionMinSize="2048"
>noCompressionUserAgents="gozilla,traviata" />
>
> address="hostname"
>secretRequired="false"
>connectionTimeout="2"
>server=" "
>maxHttpHeaderSize="8192"
>maxThreads="500"
>minSpareThreads="30"
>enableLookups="false"
>disableUploadTimeout="true"
>acceptCount="150"
>redirectPort="9443"
>scheme="https"
>secure="true"
>
>  
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
>compression="on"
>compressionMinSize="2048"
>noCompressionUserAgents="gozilla,traviata" />
>
>
>
> Shawn Beard
> Sr. Systems Engineer
> BTS
> +1-515-564-2528
>
> -Original Message-
> From: Mark Thomas 
> Sent: Thursday, September 17, 2020 3:19 AM
> To: users@tomcat.apache.org
> Subject: Re: tomcat warnings. [EXTERNAL]
>
> ** CAUTION: External message
>
>
> On 16/09/2020 20:45, Beard, Shawn wrote:
> >  >protocol="HTTP/1.1"
> >connectionTimeout="2"
> >Server=" "
> >maxHttpHeaderSize="8192"
> >maxThreads="500"
> >minSpareThreads="30"
> >enableLookups="false"
> >disableUploadTimeout="true"
> >acceptCount="150"
> >redirectPort="9444"
> >
> compressibleMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/xml,application/x-javascript,application/javascript,application/x-font-woff,application/font-woff,application/pdf"
> >compression="on"
> >compressionMinSize="2048"
> >   

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi Remy,

On Tue, Sep 15, 2020 at 6:16 PM Rémy Maucherat  wrote:

> On Tue, Sep 15, 2020 at 5:08 PM Martin Grigorov 
> wrote:
>
> > Hi Mark,
> >
> > On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas  wrote:
> >
> > > On 15/09/2020 12:46, Martin Grigorov wrote:
> > > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov <
> mgrigo...@apache.org>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I am running some load tests on Tomcat and I've noticed that when
> > HTTP2
> > > is
> > > >> enabled the throughput drops considerably.
> > > >>
> > > >> Here are the steps to reproduce:
> > > >>
> > > >> 1) Enable HTTP2, e.g. by commenting out this connector:
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
> > > >>
> > > >> 2) Download Vegeta load tool from:
> > > >> https://github.com/tsenart/vegeta/releases/
> > > >>
> > > >> 3) Run the load tests:
> > > >>
> > > >> 3.1) HTTP/1.1
> > > >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/
> "}'
> > |
> > > >> vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s
> |
> > > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json
> > > >> /tmp/http1.json | jq .
> > > >>
> > > >> 3.2) HTTP2
> > > >> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/
> > "}'
> > > |
> > > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000
> -insecure
> > > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
> > > >> -type=json /tmp/http2.json | jq .
> > > >>
> > > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0
> means
> > > >> that Vegeta will try to send as many requests as possible with the
> > > >> configured number of workers.
> > > >> I use '-insecure' because I use self-signed certificate.
> > > >>
> > > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only
> > > responses
> > > >> with code=200 .
> > > >> But for HTTP2 Tomcat starts returning such kind of errors:
> > > >>
> > > >>  "errors": [
> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
> > > GOAWAY
> > > >> and closed the connection; LastStreamID=9259,
> ErrCode=PROTOCOL_ERROR,
> > > >> debug=\"Stream [9,151] has been closed for some time\"",
> > > >> "http2: server sent GOAWAY and closed the connection;
> > > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151]
> has
> > > been
> > > >> closed for some time\"",
> > > >> "Get \"https://localhost:8443/examples/\": http2: server sent
> > > GOAWAY
> > > >> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
> > > >> debug=\"Stream [49] has been closed for some time\""
> > > >>   ]
> > > >>
> > > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
> > >
> > > That indicates that the client has sent a frame associated with a
> stream
> > > that the server closed previously and that that stream has been removed
> > > from the Map of known streams to make room for new ones. See
> > > Http2UpgardeHandler.pruneClosedStreams()
> > >
> > > It looks like the client is making assumptions about server behaviour
> > > that go beyond the requirements of RFC 7540, section 5.3.4.
> > >
> >
> > This is possible!
> > I've just tested with two more HTTP2 impls:
> >
> > 1) Node.js
> >
> > http2-server.js
> > ===
> > const http2 = require('http2');
> > const fs = require('fs');
> >
> > const server = http2.createSecureServer({
> > key: fs.readFileSync('/path/to/server.key'),
> > cert: fs.readFileSync('/path/to/server.crt')
> > });
> > server.on('error', (err) => console.error(

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi Chris,

On Tue, Sep 15, 2020 at 5:09 PM Christopher Schultz <
ch...@christopherschultz.net> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Martin,
>
> On 9/15/20 07:37, Martin Grigorov wrote:
> > I am running some load tests on Tomcat and I've noticed that when
> > HTTP2 is enabled the throughput drops considerably.
> >
> > Here are the steps to reproduce:
> >
> > 1) Enable HTTP2, e.g. by commenting out this connector:
> > https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f31
> 7c135d9d/conf/server.xml#L103-L112
> <https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112>
> >
> >  2) Download Vegeta load tool from:
> > https://github.com/tsenart/vegeta/releases/
> >
> > 3) Run the load tests:
> >
> > 3.1) HTTP/1.1 echo -e '{"method": "GET", "url":
> > "http://localhost:8080/examples/"}' | vegeta attack -format=json
> > -rate=0 -max-workers=1000 -duration=10s | vegeta encode >
> > /tmp/http1.json; and vegeta report -type=json /tmp/http1.json | jq
> > .
> >
> > 3.2) HTTP2 echo -e '{"method": "GET", "url":
> > "https://localhost:8443/examples/"}' | vegeta attack -format=json
> > -http2 -rate=0 -max-workers=1000 -insecure -duration=10s | vegeta
> > encode > /tmp/http2.json; and vegeta report -type=json
> > /tmp/http2.json | jq .
>
> You are using HTTP with 1.1 and HTTPS with h2. Maybe you are seeing
> CPU slowdown for the (probably double encryption) taking place on the
> same host?
>

I've tested against HTTPS without H2 - it makes 13K reqs/sec here, i.e. 2-3
K less than HTTP.
The above numbers are when I use the examples' index.html page as a target.
When I use

=
package load.servlet;

import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

@WebServlet(urlPatterns = PlainTextServlet.URL_PATTERN)
public class PlainTextServlet extends HttpServlet {

  static final String URL_PATTERN = "/servlet/plaintext";
  private static final String CONTENT_TYPE = "text/plain;charset=UTF-8";
  private static final byte[] CONTENT = "Hello
world!".getBytes(StandardCharsets.UTF_8);
  private static final int CONTENT_LENGTH = CONTENT.length;

  @Override
  protected void doGet(final HttpServletRequest req, final
HttpServletResponse resp) throws IOException {
resp.setContentType(CONTENT_TYPE);
resp.setContentLength(CONTENT_LENGTH);
resp.getOutputStream().write(CONTENT);
  }
}
=

I can get 33-35 K reqs/sec with HTTP



>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9gyyIACgkQHPApP6U8
> pFiNUxAAnUQVU73nNitIyn6zD9t4JfkLIv/AKTlds4/W/p6TRtIQgTX7nIJjGDfw
> BOKznCHmieMJon4rMZ3d8GFhmUP8CawQlJHpqABeITBzLZZ5x9fuOf22G6HJb3r+
> +k0qDoKzFTWlJuWLwaLZHy6fO9ugi4OPAW0G0efa2T6sTDZzImGnjmoZ5PWBExoz
> mXmYWnZeP7R+3QkAWUYArJh9yPEJyIb9nFX1YKZ1l5Erzrn0F9uEYFgWT/UkQoKM
> L65AMh/qEvzJhP2wHOLm4NfAiNO4OgTmo+nm4F/SIMGFNURPFi2sl/jTUHVAzEa4
> mAqlJqX1swimyjjsunlfhbU/bApvVFsYSPuSYcZmLN1lkmaQOAuWHnZdd4e9h+tt
> rhoKXipk8OairYzwQsPVnzCTHaiAhOXJ3MSE966YwlvhSMOoqDsN3y7ySrboresD
> iC0cDo+43/wR3IQlOJYFxcFX+tI2Y29ZjrX/IwnJXuVyU095YZWmRFC2JgRfzBtI
> toM2ofpqnSBaS22ZBTbqp+q1QxRZfC3r0vuvuiXK620QRcbk1Ya0+U17LOIEYnuY
> URY94kL80upiADQMIdryq4ubRAma2t0s5c6JuO/QqsXVjJfawlRGQA5arORgfE2J
> yDCscyyFCHitEGTglIJUXW/KfFPtraWnON3TSCm7dQ55EmInxpc=
> =76Zf
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Low throughput with HTTP2

[Martin Grigorov]

Hi Mark,

On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas  wrote:

> On 15/09/2020 12:46, Martin Grigorov wrote:
> > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov 
> > wrote:
> >
> >> Hi,
> >>
> >> I am running some load tests on Tomcat and I've noticed that when HTTP2
> is
> >> enabled the throughput drops considerably.
> >>
> >> Here are the steps to reproduce:
> >>
> >> 1) Enable HTTP2, e.g. by commenting out this connector:
> >>
> >>
> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
> >>
> >> 2) Download Vegeta load tool from:
> >> https://github.com/tsenart/vegeta/releases/
> >>
> >> 3) Run the load tests:
> >>
> >> 3.1) HTTP/1.1
> >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}' |
> >> vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s |
> >> vegeta encode > /tmp/http1.json; and vegeta report -type=json
> >> /tmp/http1.json | jq .
> >>
> >> 3.2) HTTP2
> >> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/"}'
> |
> >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure
> >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
> >> -type=json /tmp/http2.json | jq .
> >>
> >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means
> >> that Vegeta will try to send as many requests as possible with the
> >> configured number of workers.
> >> I use '-insecure' because I use self-signed certificate.
> >>
> >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only
> responses
> >> with code=200 .
> >> But for HTTP2 Tomcat starts returning such kind of errors:
> >>
> >>  "errors": [
> >> "Get \"https://localhost:8443/examples/\": http2: server sent
> GOAWAY
> >> and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR,
> >> debug=\"Stream [9,151] has been closed for some time\"",
> >> "http2: server sent GOAWAY and closed the connection;
> >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has
> been
> >> closed for some time\"",
> >> "Get \"https://localhost:8443/examples/\": http2: server sent
> GOAWAY
> >> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
> >> debug=\"Stream [49] has been closed for some time\""
> >>   ]
> >>
> >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
>
> That indicates that the client has sent a frame associated with a stream
> that the server closed previously and that that stream has been removed
> from the Map of known streams to make room for new ones. See
> Http2UpgardeHandler.pruneClosedStreams()
>
> It looks like the client is making assumptions about server behaviour
> that go beyond the requirements of RFC 7540, section 5.3.4.
>

This is possible!
I've just tested with two more HTTP2 impls:

1) Node.js

http2-server.js
===
const http2 = require('http2');
const fs = require('fs');

const server = http2.createSecureServer({
key: fs.readFileSync('/path/to/server.key'),
cert: fs.readFileSync('/path/to/server.crt')
});
server.on('error', (err) => console.error(err));

server.on('stream', (stream, headers) => {
// stream is a Duplex
stream.respond({
'content-type': 'text/plain; charset=utf-8',
':status': 200
});
stream.end('Hello world!');
});

server.listen(18080);
===

run with: node http2-server.js

Runs fine with -rate=0 and gives around 8K reqs/sec

2) Rust

Cargo.toml
===
[package]
name = "my-http2-server"
version = "0.0.1"
publish = false
authors = ["Martin Grigorov "]
license = "MIT/Apache-2.0"
description = "Load test HTTP/2 "
repository = "https://github.com/martin-g/http2-server-rust;
keywords = ["http2"]
edition = "2018"

[dependencies]
actix-web = { version = "3", features = ["openssl"] }
openssl = { version = "0.10", features = ["v110"] }
===

src/main.rs
===
use actix_web::{web, App, HttpRequest, HttpServer, Responder};
use openssl::ssl::{SslAcc

Re: Low throughput with HTTP2

[Martin Grigorov]

On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov 
wrote:

> Hi,
>
> I am running some load tests on Tomcat and I've noticed that when HTTP2 is
> enabled the throughput drops considerably.
>
> Here are the steps to reproduce:
>
> 1) Enable HTTP2, e.g. by commenting out this connector:
>
> https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112
>
> 2) Download Vegeta load tool from:
> https://github.com/tsenart/vegeta/releases/
>
> 3) Run the load tests:
>
> 3.1) HTTP/1.1
> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}' |
> vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s |
> vegeta encode > /tmp/http1.json; and vegeta report -type=json
> /tmp/http1.json | jq .
>
> 3.2) HTTP2
> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/"}' |
> vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure
> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
> -type=json /tmp/http2.json | jq .
>
> As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means
> that Vegeta will try to send as many requests as possible with the
> configured number of workers.
> I use '-insecure' because I use self-signed certificate.
>
> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only responses
> with code=200 .
> But for HTTP2 Tomcat starts returning such kind of errors:
>
>  "errors": [
> "Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
> and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR,
> debug=\"Stream [9,151] has been closed for some time\"",
> "http2: server sent GOAWAY and closed the connection;
> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has been
> closed for some time\"",
> "Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
> debug=\"Stream [49] has been closed for some time\""
>   ]
>
> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s
>
>
> All the access logs look like:
>
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
> 1126
>
> i.e. there are no error codes, just 200.
> Vegeta reports the error with status code = 0. I think this just means
> that it didn't get a proper HTTP response but just TCP error.
> There are no errors in catalina.out.
>
> Are there any settings I can tune to get better throughput with HTTP2 ?
>
> Tomcat 10.0.0-M8.
>

Forgot to mention that I've also tested with JMeter +
https://github.com/Blazemeter/jmeter-http2-plugin but there it fails with
OOM if I use more than 2000 virtual users. Increasing the memory still does
not give such good results as Vegeta for HTTP/1.1. Also JMeter uses
sequential model.

For comparison, I've also tested with a simple Golang based HTTP2 server:

http2-server.go:
==
package main

import (
"fmt"
"log"
"net/http"
"os"
)

func main() {

port := 8080
if port == "" {
  log.Fatal("Please specify the HTTP port as environment variable, e.g.
env PORT=8081 go run http-server.go")
}

tls_root := "/path/to/certs/"
srv := {Addr: ":" + port, Handler: http.HandlerFunc(handle)}
log.Fatal(srv.ListenAndServeTLS(tls_root + "server.crt", tls_root +
"server.key"))
}

func handle(w http.ResponseWriter, r *http.Request) {
//log.Printf("Got connection: %s", r.Proto) // prints HTTP/2.0
fmt.Fprintf(w, "Hello World")
}
==

Here Vegeta makes around 13K reqs/sec without error responses.

To run this app do: go run http2-server.go


> Regards,
> Martin
>

Low throughput with HTTP2

[Martin Grigorov]

Hi,

I am running some load tests on Tomcat and I've noticed that when HTTP2 is
enabled the throughput drops considerably.

Here are the steps to reproduce:

1) Enable HTTP2, e.g. by commenting out this connector:
https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112

2) Download Vegeta load tool from:
https://github.com/tsenart/vegeta/releases/

3) Run the load tests:

3.1) HTTP/1.1
echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}' |
vegeta attack -format=json  -rate=0 -max-workers=1000 -duration=10s |
vegeta encode > /tmp/http1.json; and vegeta report -type=json
/tmp/http1.json | jq .

3.2) HTTP2
echo -e '{"method": "GET", "url": "https://localhost:8443/examples/"}' |
vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure
-duration=10s | vegeta encode > /tmp/http2.json; and vegeta report
-type=json /tmp/http2.json | jq .

As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means that
Vegeta will try to send as many requests as possible with the configured
number of workers.
I use '-insecure' because I use self-signed certificate.

On my machine I get around 14-15K reqs/sec for HTTP1.1 with only responses
with code=200 .
But for HTTP2 Tomcat starts returning such kind of errors:

 "errors": [
"Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR,
debug=\"Stream [9,151] has been closed for some time\"",
"http2: server sent GOAWAY and closed the connection;
LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has been
closed for some time\"",
"Get \"https://localhost:8443/examples/\": http2: server sent GOAWAY
and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR,
debug=\"Stream [49] has been closed for some time\""
  ]

when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s


All the access logs look like:

127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126
127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200
1126

i.e. there are no error codes, just 200.
Vegeta reports the error with status code = 0. I think this just means that
it didn't get a proper HTTP response but just TCP error.
There are no errors in catalina.out.

Are there any settings I can tune to get better throughput with HTTP2 ?

Tomcat 10.0.0-M8.

Regards,
Martin

Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys

[Martin Grigorov]

hi,

On Tue, Sep 15, 2020 at 8:20 AM Pratik Shrestha  wrote:

> Hi Guys,
>
> Just wanted to know if anyone found an idea on fixing it or a workaround.
>

Did you find what is the expected behavior by Qualis ?


>
> Thanks
>
> Pratik.
>
> On Fri, Aug 28, 2020 at 10:46 AM Pratik Shrestha 
> wrote:
>
> > Hi Chris
> >
> >
> >
> >
> > *This wasn't the case for httpd for many years. I don't know what itdoes
> > these days, but it used to reply with a nice "400 Bad Request"error just
> > like Tomcat is doing. The difference is that httpd has richconfiguration
> > options to allow you to override that behavior. *
> >
> > Correct. By default HTTP also gives 400 Bad Request message. But we can
> > override this behavior by using ErrorDocument directive to redirect to
> > https URL. Currently in Tomcat this feature is not available. Tomcat has
> > RewriteValve but it does not work in this case.
> >
> > On Fri, Aug 28, 2020 at 12:30 AM Christopher Schultz <
> > ch...@christopherschultz.net> wrote:
> >
> >> -BEGIN PGP SIGNED MESSAGE-
> >> Hash: SHA256
> >>
> >> Merka,
> >>
> >> On 8/27/20 06:32, Phoenix, Merka wrote:
> >> > I think what the Qualys scan is trying to flag is that the server
> >> > (Tomcat) is listening for both secured and unsecured traffic on
> >> > the _same_ TCP port when the server should be listening for just
> >> > one of the two options (unsecured or secured), not both.
> >> This actually might be a semi-legitimate test. If the client says "Our
> >> policy is to only communicate using encrypted connections" and the
> >> test says "well, here's a non-encrypted connection right here!" then
> >> it makes some small amount of sense. In that case, it's all driven by
> >> policy, and the policy can say "we have a carve-out for TLS handshake
> >> failures" and then allow that particular test to pass.
> >>
> >> > The error message returned by the Tomcat service, while certainly
> >> > helpful to the remote client, is returning more information than
> >> > it should (from a security-viewpoint).
> >> Not really.
> >>
> >> > If the default behavior for Tomcat is to return this "helpful"
> >> > message for misconfigured clients, would it be reasonable for the
> >> > Connector to have an option (attribute) for turning off this
> >> > feature and simply reject with a TLS error any unsecured traffic on
> >> > the port? This would address the security concern without imposing
> >> > too much "bloat" to the Tomcat side.
> >> I think this might actually be better/easier than implementing a
> >> redirect. It's a simple flag that says "return nice errors on
> >> plaintext-over-TLS connection attempts" (or similar) and the only
> >> thing that changes is that we STOP trying to be nice to the client if
> >> the setting is set to "fail" versus "be-nice".
> >>
> >> > For most other web servers (Apache httpd, NGINX, etc.) that offer
> >> > https, the normal behavior is that when an http client tries to
> >> > speak http to server expecting https, the client sees some garbled
> >> > text (the server's TLS response to the connection attempt).
> >> This wasn't the case for httpd for many years. I don't know what it
> >> does these days, but it used to reply with a nice "400 Bad Request"
> >> error just like Tomcat is doing. The difference is that httpd has rich
> >> configuration options to allow you to override that behavior.
> >>
> >> - -chris
> >> -BEGIN PGP SIGNATURE-
> >> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
> >>
> >> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7ZkACgkQHPApP6U8
> >> pFgOng/8DtMJkQc+MYLRm1iuCD7GZ2f2S7oQ59vFeCGAbmkiUjESvQ42QRGqIMy8
> >> 47giFc3ERm84DxyyyU7O/YDFxinOnCrC/v9A6RzpYKlZBSOq9Oy6732xTUAqGGIw
> >> +3QGPXvyjE2Vcg1iavW+7cUKN1Q9R1NGcDRRBpBL0KRQMA3NV9pxGU71/In8TQvi
> >> VZ51f7VTNXaJ2l+w6G23XBJkKQk3csFixevVzr4Xr56FLfqPxUc3m6QNu4BaHmgb
> >> c95hceV/N7tkR9yHdaRtahpZq0lhGqXbNXfqjf7kElSkRmeAZ3MSsdnFD4fBHThn
> >> xvz204xffSE71Z4W24W9gx23+Bg0y2EfPRo1CWC93rEvNRKMK6ILzLczTRA0w6QW
> >> 9zP1XC+VwC25LQOGFDgFukQVupPYiMoNSb6DRey5ZUhur6v25nevwbhM0QsAm/oO
> >> oZpreKaUMy+ZoixwGhaZ+UFiZRav7DRLSj85BjK9PqcP4VdPzFR9MarvMqLPxRoq
> >> YxL/jNet4L+29Z2tDkZv4gfGJqI7oWkfXUsBFBjj5JgXrqE94Q8PzAK87pLeU80y
> >> p3IL4krovHbu01j1fE3/aDotEvBu/wxWTCWze9+vL09a82PuTT2pihyCVqFuP9rS
> >> kP0DtVTfbaUMyD2dryjyw4q1NdLYht4y/HHkyU/3cPopCbxEopU=
> >> =4F4z
> >> -END PGP SIGNATURE-
> >>
> >> -
> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>
> >>
>

Re: Tomcat Processing Timer Question

[Martin Grigorov]

Hi,

On Sat, Sep 12, 2020, 02:57 Eric Robinson  wrote:

> I'm not sure what you mean by measuring at the load balancer level. We're
> using the jasper logs and those only exist on the tomcat server itself. I
> must be misunderstanding your meaning.
>

He meant to use the LB's logs for the same.
What software do you use for load balancing?


> Get Outlook for Android
>
> 
> From: Christopher Schultz 
> Sent: Thursday, September 10, 2020 3:11:43 PM
> To: users@tomcat.apache.org 
> Subject: Re: Tomcat Processing Timer Question
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Eric,
>
> On 9/10/20 15:29, Eric Robinson wrote:
> > Chris --
> >
> >
> >> You should also look at worker-thread availability. When you see
> >>  these "high latency" (which is usually a term reserved for I/O
> >> characterization) events, do you have:>> 1. Available worker
> >> threads (from the executor thread pool) 2. Any other
> >> shared/limited resource (e.g. DB connection pool)
> >>
> >
> > Good thought. I should mention that the hosted application is
> > canned, and is the same for all tomcat instances, with only minor
> > variations in version between them. User workflow is also similar.
> > Over the years we've developed a good feel for expected
> > performance and resource utilization based on the user count per
> > instance. So when one instance exhibits anomalous performance, we
> > tend to go right to networking issues.
> >
> >> Also, are you seeing the otherwise unexpected slowness on each
> >> Tomcat node, or are you seeing it at the
> >> load-balancer/multiplexer level?
> >>
> >
> > We run multi-tenanted servers, with many instances of tomcat on
> > each server. We've never seen issues at the load-balancer.
>
> What I mean is, are you measuring the request at the Tomcat level, or at
> the load-balancer level? If you are watching at the lb, then your lb
> might pick a "busy" Tomcat and the request has to "wait in line" before
> processing even begins. If you sample at the Tomcat level, you'll see no
> discernible slowdown because the time "waiting in line" does not count.
>
> > Very occasionally, there might be a problem at the server level.
> > When that happens, all instances on that server may become
> > sluggish. What I'm talking about in this thread are cases where
> > only one instance on a server is showing slowness in its jasper
> > logs. Also, we typically do not see the same slowness when we test
> > the application locally from the same network. I've had my eye on
> > TCP retransmits as a possible culprit for a while, but I just
> > didn't know for sure if my understanding of the tomcat processing
> > timer is correct.
> I hope we've cleared that up for you, then.
>
> You might also want to read about "buffer bloat" if you aren't already
> familiar with that term.
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
>
> iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9aiH4ACgkQHPApP6U8
> pFiFfBAAvuUbRXK+iDDy7lLsw6eplMFrXXDbkxzwtSNafvdGlDWmcPWwdazZwhQ+
> TJ0pzkUwf3/RBslu/oORJYelYKhpUJLodj0Y85ZtbuKBcU2JpKk1uueJ/aqnmVFK
> 9yep3ReYdggEXQ3JNb1VeI4ASdEhFWoFw8pc6DAcJZ4K2JaUtGKrtoWG8n+oEXos
> kmthl9Dm9ge3edLimd7TPTx11iODi6pX3ddJ+uRh7qmvXZp4wVyX8W+hkKiOhUQM
> hokUd8RruXQm6wut5m+JSO6eLHqkKUBiLspzlz1x/Y4cuaqAlC8Pl5y9NFTuLK3e
> gFJeDmBUthN2y5h9KNKW5r+Gf9bKpuv1+kw7CIaNoFv2JxCGTmfL3VKM+Bp/rh7J
> 1SbshsTW6ffo5hKRNJUJKvxry3uUvzrss0AYe338tJ1QA+sHuXHsN8ZVtY3b+51O
> HBOYf3pgIPsSd6zXkjaSRoOAhVc9G5sbJHx8ycQt+yAyVvXEUwrqeeRbsJeADk2s
> reaizm9WvO2kHSqP93ANNYe1QJ+rw9b5og0uoCE8x9eO+czRHbJ7LFF6/rvX+6Pn
> TIYB7AHyV8P3PHpHtBGIgaNfnvIYbqx/hzxJpLlpNEcS2zARfi1YCnuNtbiH0KU/
> AKkBx5FnZvwclCA3qK2oqBnSEcBUFz2yobq4wAy//qwgL2gEFNc=
> =mcpm
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
> Disclaimer : This email and any files transmitted with it are confidential
> and intended solely for intended recipients. If you are not the named
> addressee you should not disseminate, distribute, copy or alter this email.
> Any views or opinions presented in this email are solely those of the
> author and might not represent those of Physician Select Management.
> Warning: Although Physician Select Management has taken reasonable
> precautions to ensure no viruses are present in this email, the company
> cannot accept responsibility for any loss or damage arising from the use of
> this email or attachments.
>

Re: How to Set Content Security Policy headers in Tomcat 8.5.x

[Martin Grigorov]

Hi,

On Wed, Sep 9, 2020 at 8:54 PM Nitin Kadam  wrote:

> Hi All,
>
> Need to set the *Content security policy* header for Tomcat Web server
> (8.5..x) which hosted on Windows server 2012, As per the internal security
> team same is not a complaint
> can you please help me setting CSP filters for my Tomcat application
> hosted on windows server.
>

You can use javax.servlet.Filter to add such custom headers.
See
https://github.com/apache/tomcat/blob/53c304ad1f65a09c921c40e03a115de438f6c68a/java/org/apache/catalina/filters/HttpHeaderSecurityFilter.java
for
inspiration.
More about Filters you can read in the web, e.g.
https://www.tutorialspoint.com/servlets/servlets-writing-filters.htm


>
> below the screenshot from securityheaders.com
>
> [image: image.png]
>
> --
> Regards
> Nitin Kadam
>
>

Re: HTTP2: Connections abruptly closed by sending GOAWAY

[Martin Grigorov]

Hi,

On Wed, Sep 9, 2020 at 3:31 PM Arshiya Shariff
 wrote:

> Thank you for the response Mark .
>
>
>
> Hi All,
>
>
>
> Can you please help us understand this behavior .
>
> The following is the sequence of events that is happening for a few
> streams .
>
> Initially when the stream is opened , it processes fine , but beyond some
> time tomcat does not process the headers, for which the client is sending
> RST_STREAM (ERROR: NO_ERROR) and then tomcat closes the stream with GOAWAY
> (An error occurred during processing that was fatal to the connection).
>
>
>
>
Your images didn't make it thru the mailing list.
https://markmail.org/message/c5pz2bmuaeodrlhr
Please use some image bin and give us the url.

> Please help us analyze under which cases this happens .
>
> Embedded tomcat:9.0.22
>
>
>
> Thanks and Regards
>
> Arshiya Shariff
>
>
>
>
>
> -Original Message-
> From: Mark Thomas 
> Sent: Monday, September 7, 2020 10:21 PM
> To: users@tomcat.apache.org
> Subject: Re: HTTP2: Connections abruptly closed by sending GOAWAY
>
>
>
> On 07/09/2020 09:29, Arshiya Shariff wrote:
>
> > Hi All,
>
> > Tomcat is closing connections abruptly by sending GOAWAY with reason
>
> > Connection [5309], Stream [57,359], An error occurred during processing
> that was fatal to the connection .
>
> >
>
> > Just trying to understand in what scenarios this happens. Can you please
> help us.
>
>
>
> An unhandled InterruptedIOException during I/O.
>
>
>
> I/O exception writing an intermediate 100 response for a request with an
> expectation.
>
>
>
> I/O exception trying to initiate a push request.
>
>
>
> I/O exception committing, flushing or closing a stream.
>
>
>
> So, generally, any I/O exception that indicates the connection between the
> client and the server is broken.
>
>
>
> Mark
>
>
>
> -
>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>

Re: Release date of latest Tomcat version - 9.0.38

[Martin Grigorov]

Hi,

On Wed, Sep 2, 2020, 17:46 Arshiya Shariff
 wrote:

> Hi ,
> Thank you for your response .
> Is there a way I can track when the version is released for Voting (or)
> will I get notified in the Users list ?
>

You need to subscribe to the dev@ mailing list.



> Thanks and Regards
> Arshiya Shariff
>
> -Original Message-
> From: Mark Thomas 
> Sent: Wednesday, September 2, 2020 12:03 AM
> To: users@tomcat.apache.org
> Subject: Re: Release date of latest Tomcat version - 9.0.38
>
> On 01/09/2020 14:01, Christopher Schultz wrote:
> > Arshiya,
> >
> > On 9/1/20 08:13, Arshiya Shariff wrote:
> >> Hi all,
> >
> >> The following reported issue - "HTTP/2 Stream.receivedData method
> >> throwing continuous NullPointerException in the logs" has been fixed
> >> in the latest tomcat.
> >> https://bz.apache.org/bugzilla/show_bug.cgi?id=64671
> >
> >> Can you please share us the release date of tomcat version 9.0.38 .
> >> We are waiting for the release dates so we can plan accordingly.
> > There are no promises about release schedule.
> >
> > Mark, however, has fairly consistently been rolling releases around
> > the beginning of each month. If you read the developers list, you'll
> > see there was a conversation started last week about the next batch of
> > releases.
>
> I'm expecting to tag the next round of releases in a few days. I am
> currently waiting for the Commons Daemon 1.2.3 release (it was delayed by
> an issue with the code signing service) and the Tomcat Native 1.2.25
> release (vote should hopefully complete shortly).
>
> If the Commons Daemon release slips then the Tomcat release will probably
> slip.
>
> > Once a release candidate has been announced (look for [VOTE] threads),
> > you can help by testing the release:
>
> Big +1
>
> >   1. Run the test suite (ant test)
> >   2. Run the release-candidate on your own application
> >
> > Your vote is not binding, but if you find a problem, we will likely
> > stop the release to fix it.
>
> Indeed. We've done that a few times. Much better to test the release
> during the vote (when we can fix it and re-roll the release) rather than
> once it is released (when you have to wait for the next release in ~1
> month). The number of times people have downloaded a new release and found
> an issue that they could have caught if they tested during the vote...
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

Re: Tomcat 9 : Unable to specify wildcard care name in Host

[Martin Grigorov]

On Tue, Aug 25, 2020 at 3:01 PM Tom Chiverton  wrote:

> I'd like to propose some changes to the docs, but can not locate their
> source to submit a request.
>

https://github.com/apache/tomcat/blob/master/webapps/docs/config/host.xml
This is for Tomcat 10.x (master branch).


>
> For instance, in https://tomcat.apache.org/tomcat-9.0-doc/config/host.html
>
> "If the name takes the form of |*.domainname| (e.g. |*.apache.org|) then
> it will be treated as a match for any host in that domain unless a host
> that has an exactly matching name is found."
>
> would be clearer as
>
> "The name can not contain a wildcard (e.g. |*.apache.org|), this is only
> valid in an Alias."
>
> And then in
> https://tomcat.apache.org/tomcat-9.0-doc/config/host.html#Host_Name_Aliases
>
> "Aliases may also use the wildcard form (|*.domainname|) permitted for
> the *name* attribute of a *Host*."
>
> to
>
> "Aliases may also use the wildcard form (|*.domainname|), unlike for the
> name attribute of a Host**."
>
> On 24/08/2020 13:14, Tom Chiverton wrote:
> > Am I mis-reading the docs ?
>
>
> __
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> __

Re: Tomcat v9 - Insecure transport vulnerability reported by Qualys

[Martin Grigorov]

Hi,

On Wed, Aug 26, 2020 at 7:53 AM Pratik Shrestha  wrote:

> Thanks for reply,
>
> Hi Peter - it complains on port 8443 which belongs to Tomcat.
>
> Hi Mark - Yes. making HTTP request on HTTPS is wrong. But this security
> vulnerability is given to us by Qualys scan. It tries to post plain HTTP
> request on HTTPS port and then gets error message "Bad Request. This 
> combination
> of host and port requires TLS." which is security loop hole for Qualys.
> This is behaviour of Apache HTTP server also. But in Apache though, we can
> get rid of this by using "ErrorDocument 400" directive. Do we have similar
> in Tomcat? I have already tried using
>
> 
>400
>/error.jsp
>  
>

This won't work because Tomcat stops the request earlier and doesn't pass
it to your application.
I haven't tried it but it may work with a custom Valve, extending
ErrorReportValve.


>
> Not sure, but my idea was to add redirect code on error.jsp page. But
> above never works. It never reaches error.jsp page. Just sticks in default
> error message page mentioned above.
>
> Btw..you can see the result from Qualys attached.
>

What is the desired behavior expected by Qualys ?
Because at the moment Tomcat returns a text/html error page and you try to
"fix" it by returning a custom text/html error page. I don't see how this
will change the Qualys report.


>
> Thanks again guys for getting back.
>
> Regards,
> Pratik
>
> On Tue, Aug 25, 2020 at 5:36 PM Mark Thomas  wrote:
>
>> On 25/08/2020 11:14, Pratik Shrestha wrote:
>> > Hi all,
>> >
>> > Tomcat version: 9.0.37
>> >
>> > Our website is running on Tomcat. We did Qualys vulnerability scan on
>> our
>> > site. Scan shows below vulnerability.
>> >
>> > Insecure transport
>> > Group: Information Disclosure
>> > CWE CWE-319
>> > OWASP A3 Sensitive Data Exposure
>> > WASC WASC-4 INSUFFICIENT TRANSPORT LAYER PROTECTION
>> >
>> > Please note
>> > 1. HTTP port is not enabled.
>> > 2. We have only opened HTTPS port 8443. But when we connect this HTTPS
>> port
>> > with HTTP (http://www.oursite.com:8443/), we get an error "Bad
>> Request. This
>> > combination of host and port requires TLS."
>> > 3. Due to the above error message, we get this vulnerability error from
>> > Qualys.
>> > 4. We have already enabled HSTS.
>> > 5. We have enabled Rewrite Valve also to direct all HTTP to HTTPS. But
>> it
>> > never works. It is like, Tomcat doesn't care about Rewrite or HSTS. It
>> just
>> > finds someone is accessing HTTPS port with HTTP protocol and then just
>> > throws error 400 'Bad Request'
>> > 6. Note that Tomcat version 7 used to send the error 'ERR_EMPTY_RESP'
>> which
>> > should still be okay.
>> >
>> > We already tried to find the fix for this issue on the web but in vain.
>> >
>> > Kindly help if anyone has found a way to fix it.
>>
>> Fix what?
>>
>> If you make an HTTP request to an HTTPS port, Tomcat provides a helpful
>> error message.
>>
>> I don't see any security issues here.
>>
>> (And before anyone claims the request sent in the clear is insecure I'll
>> point out that the request is sent in the clear irrespective of whether
>> Tomcat responds with an HTTP/1.1 clear text error message or a cryptic
>> TLS failure).
>>
>> Mark
>>
>> -
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org