Re: Virtual Thread Configuration In Tomcat 11
Chris, Great post earlier today! This is a super interesting topic to me. You can find the performance testing results located here: http://ec2-18-188-185-212.us-east-2.compute.amazonaws.com:8080/web-report/ I did 10 runs with 1000 threads with a ramp up time of 3 seconds for a duration of 200 seconds. Ignore the 11th run. This is a really simple REST application with Spring. It just makes an insert into a MySQL table. When I get a moment I will put my code into GitHub, but again, there is not much to it. You can find a very similar Spring Boot example with embedded Tomcat here: https://medium.com/@anil.java.story/embracing-virtual-threads-in-spring-boot-4140d3b8a5a But he uses JDK 20 with --enable-preview turned on. The mistake I think I made was that the inserts are not really blocking because they happen so quickly. Maybe because I did not put enough load in the test? I have no synchronized blocks in my code. So, it appeared to me that the inserts were taking twice the amount of time with virtual threads. I think that extra time is just overhead of pinning virtual threads to a platform thread and managing the ForkJoinPool. What is going to happen is that people are going to “flip the switch”…useVirtualThreads=”true”. They will find that performance will not be as good as using an operating system thread and abandon virtual threads because their code does not block. So, I need to come up with an example where my code blocks and redo the perf test and prove this out. I don’t know. Regards, William Crowell This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Virtual Thread Configuration In Tomcat 11
William,
On 8/24/23 09:50, William Crowell wrote:
I did some performance testing with virtual threads on Apache Tomcat
11.0.0-M10 and JDK 21 (21+35-2513). I have a simple REST service
using Spring 6.0.11 that does an insert into MySQL 8.0.32.
I have 3 separate boxes all running Rocky Linux 9.2 on AWS
(t3a.xlarge which is 4 vCPUs and 16GiB RAM):
1) An application server running Tomcat 11.0.0-M10 with JDK 21
2) MySQL 8.0.32
3) JMeter 5.6.2
I know that JDK 21 is nowhere near GA, but I got some interesting
results. I did 10 test runs with useVirtualThreads=“true”, and 10
test runs without virtual threads (the default). Each run used 1000
threads in JMeter for about 3 minutes and 20 seconds for a ramp up
time of 3 seconds.
What I found was using platform threads performed at least twice as
fast as virtual threads. Maybe I am interpreting the results wrong,
but this is what I found consistently across each test.
Again, I realize that performance is not the only goal of virtual
threads. Just my observations.
This is "interesting", for some values of interesting.
I'd be interested in what the application was doing. Virtual Threads
seem to be aimed at things like Web Application Containers where there
is a lot of wasted time in a thread's execution waiting on I/O
(blocking). If you ignore the application's workload (hah!), the
application server is almost entirely occupied with (a) reading the
request and (b) writing the response. There is almost no "computation
time" and so Virtual Threads make a whole lot of sense.
Once inside the application, that may not be true.
From what little reading I have done on Virtual Threads, there are some
poison pills in there depending upon application design. For example, if
your thread blocks while inside a synchronized{} block, then your
Virtual Thread is "pinned" and will NOT be un-mounted from the platform
thread during the blocking operation. So all of that high-speed
context-switching and magic that VT is supposed to provide goes
completely out the window when you used to have 200 threads with maybe 8
of them actually running at once but swapping-out preemptively, but now
you have 200 virtual threads with 8 of them running at once, but of the
8 running at once, several of them are stopped dead unable to make any
progress.
At $work, we have plenty of read-through cache routines like this:
private Object somethingBigLock = new Object();
private SomethingBig big = null;
public Object getSomethingBigFromDatabase() {
synchronized(somethingBigLock) {
if(null == big) {
big = reallyGetSomethingBigFromDb(); // Non-trivial
}
return big;
}
}
Even if reallyGetSomethingBigFromDb() is mostly blocking on I/O -- e.g.
waiting for the db to respond to a query -- it will consume one of your
platform threads and lock it up, leaving fewer platform threads
available to do the other work.
Without a critical analysis of your application and the libraries it
uses, it's hard to tell if what you are observing is that "virtual
threads are not as fast as regular threads" or "your application is an
anti-pattern for Virtual Threads". My guess is that your application
and/or the libraries you are using need to be re-written so they no
longer use "classic" synchronization and instead use things like
ReentrantLock for a similar purpose to play-nice with VT. Or you can
wait for the (likely) inevitable improvement of the JVM which allows for
threads holding monitors to be unmounted from platform threads. My guess
is that is in the future for the JVM.
-chris
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Virtual Threads
Mark, On 9/5/23 15:55, Mark Thomas wrote: On 05/09/2023 20:38, Christopher Schultz wrote: All, I have some questions about Virtual Threads and their use within Tomcat. Note that only Tomcat 11 currently has support for Virtual Threads when running on a version 19 or later JVM. Not quite. All current versions support virtual threads when running on Java 21 or later. Thanks for the correction. I just did a quick docs[1] search for "virtual" in Tomcat 10.x for example and I didn't see useVirtualThreads, so I assumed it wasn't in there. Maybe we need some documentation back-ports? My (admittedly limited) understanding is that the use of Virtual Threads, specifically within Tomcat, will allow Tomcat to use fewer platform threads (e.g. native OS threads) to support a larger number of JVM/application threads for request-processing. The justification for this improvement is that request-processing threads spend a lot of time in an I/O-blocked state which, when using Virtual Threads, would no longer consume a platform thread. There appears to only be one setting to enable Virtual Threads in Tomcat: or In both cases, there is only one setting to affect the number of "threads" (by any description) which the Executor will ultimately use (Connectors without an explicit Executor will create a non-shared Executor to be used internally). That setting is "maxThreads" which limits the total number of "threads" that the Executor will create. The implementation of the VirtualThreadExecutor does not seem to have any upper-bound on the number of Virtual Threads that will be created at all. I believe this means that a large number of incoming requests (i.e. a burst) will result in the creation of a large number of Virtual Threads. Without an upper-bound, we are expecting that the JVM's (virtual) thread scheduler will schedule each application thread to be mounted to a platform thread in the order it was added to the queue (essentially in request-order). The virtual thread scheduler uses a work stealing queue so it isn't quite FIFO. That sounds like nit-picking to me. I suppose if each new Virtual Thread is assigned to a platform thread when it's initially queued, things can be executed in an not-strictly-FIFO-manner, but one has to assume work loads are equal (which they likely are not) and work is spread evenly across all the platform threads (which is likely) to have a general conversation about all this. I think the work-stealing ForkJoinPool is about as close to FIFO as you can get without introducing more expensive contention in order to enforce strict FIFO while not gaining much in the way of meaningful "fairness". But it's probably worth mentioning that the queue might not be "strictly fair". What *is* fair, though -- and maybe more fair than in the past? -- is that pipelined requests have to get back in line instead of jumping the queue. I think the old blocking JIO connector would allow pipelined requests to skip the queue, but all NIO-based connectors are "more fair" than that. Before Virtual Threads were introduced, the Executor would use a queue of requests to dispatch to available (non-Virtual) threads which would use that thread until the request was complete, then return the thread to the pool. With Virtual Threads, the same thing is essentially still happening except that (a) Tomcat no longer manages the thread pool (a JVM-defined one is being used instead) and (b) requests are immediately handed a (Virtual) thread to carry their execution. I believe there are some subtle differences in how Tomcat will behave, now. As an example, if I have two applications running, say, the Tomcat Manager application and the Tomcat Examples application, without using Virtual Threads, each application's thread pools should be "fair" within the context of each application: requests are processed in the order they are received in Manager and Examples, separately. If all requests are equally "expensive" and everything is "fair", then requests to the Examples application are scheduled alongside those to the Manager application, and they can both execute simultaneously as well as separately-manage the order in which the requests are processed. The above assumes each application has a separate thread pool (which implies a separate Connector). Yes, I'm sorry, I completely skipped over the "fact" that a separate would be used for such a "priority" application such as the Manager. My example was assuming that each application had the possibility to use a separate . Once Virtual Threads are introduced, the requests are filed into a single JVM-wide thread-scheduling system where activity in one application can affect the other. Correct. Let's replace Examples with RealWorldApp, an application that is expected to be used by users. Maybe a LOT of users. Without Virtual Threads, a high number of requests to RealWorldApp will not cause starvati
Re: [External] Re: Supporting Proxy Protocol in Tomcat
On 04/09/2023 15:41, Jonathan S. Fisher wrote: Mark thank you again for your leadership and setting expectations. I'm going to commit to working on this with anyone else that wants to help with the goal of a patch by year end. I want to nail the patch with minimal rework that meets Tomcat project quality standards. To that end, I'll attempt to summarize what you expect here and if you could comment and correct my understanding that would be appreciated. It sounds like you're satisfied with the ubiquity of the Proxy protocol and that it has an RFC We'll target just implementing the latest version of the Proxy protocol We'll implement a "TrustedProxies" feature similar to what the Remote IP Valve does We'll implement a, or modify the RemoteIp, valve to be able to set the remote IP from Proxy protocol headers We'll follow the RFC spec and reject any request that does a proper Proxy protocol header I'm particularly interested in the Proxy protocol over Unix Domain Sockets, so expect to see a lot of the work focused on this, but accepting Proxy Protocol over TCP looks to be quite important from the comments on this email chain If I may ask two things: Can you summarize your desired implementation? What point in the stack should we target to implement this? See my response earlier in this thread that suggested it sits alongside SNI processing. I still think that makes sense. If during implementation you reach a different conclusion then make the case for the alternative approach on list. One thing I'm not familiar with on Tomcat is the testing expectations. If you can point to a set of unit tests and a set of integration tests and say "Do it like this" Something like (only a guide) https://github.com/apache/tomcat/blob/main/test/org/apache/tomcat/util/net/TestTLSClientHelloExtractor.java to test the implementation directly and probably something based on SimpleHttpClient see https://github.com/apache/tomcat/blob/main/test/org/apache/coyote/http11/TestHttp11Processor.java for various examples. The main thing is I suspect you'll need control of the individual bytes and SimpleHttpClient provides a reasonably simple basis for that. What we often do when we want to test things like setting remote IP addresses etc. is echo the value in the response body and then check that value in the client. Anything else on the original patch you liked/didn't like? ( https://bz.apache.org/bugzilla/show_bug.cgi?id=57830) It helps if you enable Checkstyle for your local build. It helps keep things in roughly the same coding style (we are slowly tightening up on that). Ideally, use the clean-up and formatting configurations we have for Eclipse in res/ide-support/eclipse . This is sufficiently complex that I am expecting several iterations to be required. if it is simpler for you to manage with a PR then that is fine and probably easier to work with than a patch in Bugzilla. Mark Thank you, On Tue, Aug 29, 2023 at 3:13 PM Mark Thomas wrote: On 28/08/2023 18:44, Amit Pande wrote: Oh, sure. So, what would be the best way to get some conclusion on this thread? Provide a patch for review based on the feedback provided here and in the BZ issue. https://bz.apache.org/bugzilla/show_bug.cgi?id=57830 The state of the ticket isn't updated for long. Perhaps add comments/ask the folks on user list to vote? That is more likely to irritate folks rather than encourage them to help you progress your patch. Mark Thanks, Amit -Original Message- From: Mark Thomas Sent: Monday, August 28, 2023 11:20 AM To: Tomcat Users List Subject: Re: [External] Re: Supporting Proxy Protocol in Tomcat CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you believe this is a phishing email, use the Report to Cybersecurity icon in Outlook. 28 Aug 2023 17:11:20 Amit Pande : Mark, Just checking - Did this issue get discussed in any of the core members' meeting? There are no such meetings. Discussion happens on the mailing lists. Mark Thanks, Amit -Original Message- From: Amit Pande Sent: Monday, July 31, 2023 9:29 AM To: Tomcat Users List Subject: RE: [External] Re: Supporting Proxy Protocol in Tomcat Yes, understood. Thank you for clarifying. Even I was referring to initial consensus without any timeline or approach conclusion. Thanks, Amit -Original Message- From: Mark Thomas Sent: Friday, July 28, 2023 2:48 PM To: users@tomcat.apache.org Subject: Re: [External] Re: Supporting Proxy Protocol in Tomcat On 28/07/2023 19:21, Amit Pande wrote: Thank you all for the valuable discussion on this topic. Is it okay to say that we're agreeing to adding proxy protocol support in Tomcat? I think that is a little too strong. At this point there is a proposed approach and no one is objecting but until there is an actual patch to discuss... Keep in mind that a
Re: Virtual Threads
On 05/09/2023 20:38, Christopher Schultz wrote: All, I have some questions about Virtual Threads and their use within Tomcat. Note that only Tomcat 11 currently has support for Virtual Threads when running on a version 19 or later JVM. Not quite. All current versions support virtual threads when running on Java 21 or later. My (admittedly limited) understanding is that the use of Virtual Threads, specifically within Tomcat, will allow Tomcat to use fewer platform threads (e.g. native OS threads) to support a larger number of JVM/application threads for request-processing. The justification for this improvement is that request-processing threads spend a lot of time in an I/O-blocked state which, when using Virtual Threads, would no longer consume a platform thread. There appears to only be one setting to enable Virtual Threads in Tomcat: or In both cases, there is only one setting to affect the number of "threads" (by any description) which the Executor will ultimately use (Connectors without an explicit Executor will create a non-shared Executor to be used internally). That setting is "maxThreads" which limits the total number of "threads" that the Executor will create. The implementation of the VirtualThreadExecutor does not seem to have any upper-bound on the number of Virtual Threads that will be created at all. I believe this means that a large number of incoming requests (i.e. a burst) will result in the creation of a large number of Virtual Threads. Without an upper-bound, we are expecting that the JVM's (virtual) thread scheduler will schedule each application thread to be mounted to a platform thread in the order it was added to the queue (essentially in request-order). The virtual thread scheduler uses a work stealing queue so it isn't quite FIFO. Before Virtual Threads were introduced, the Executor would use a queue of requests to dispatch to available (non-Virtual) threads which would use that thread until the request was complete, then return the thread to the pool. With Virtual Threads, the same thing is essentially still happening except that (a) Tomcat no longer manages the thread pool (a JVM-defined one is being used instead) and (b) requests are immediately handed a (Virtual) thread to carry their execution. I believe there are some subtle differences in how Tomcat will behave, now. As an example, if I have two applications running, say, the Tomcat Manager application and the Tomcat Examples application, without using Virtual Threads, each application's thread pools should be "fair" within the context of each application: requests are processed in the order they are received in Manager and Examples, separately. If all requests are equally "expensive" and everything is "fair", then requests to the Examples application are scheduled alongside those to the Manager application, and they can both execute simultaneously as well as separately-manage the order in which the requests are processed. The above assumes each application has a separate thread pool (which implies a separate Connector). Once Virtual Threads are introduced, the requests are filed into a single JVM-wide thread-scheduling system where activity in one application can affect the other. Correct. Let's replace Examples with RealWorldApp, an application that is expected to be used by users. Maybe a LOT of users. Without Virtual Threads, a high number of requests to RealWorldApp will not cause starvation of requests to (maybe the more important, at least for admins) the Manager. Once Virtual Threads are introduced, a limitless number of requests can be queued in front of a request to Manager, which can experience starvation. While Tomcat did not previously implement any specific priority-queuing of requests, the use of separate Executors for each application effectively provided that kind of thing. Yes, each Executor can be configured separately to either use Virtual Threads or not, and so Manager can be configured to NOT use Virtual Threads while RealWorldApp can be configured to use Virtual Threads and the balance is "restored". But it is no longer possible to have RealWorldApp and RealWorldApp2 and Manager all with equally probable request-scheduling when using Virtual Threads. You can pick some subset of applications to get (essentially) priority by NOT using Virtual Threads, but the whole set of applications running in a single JVM will share a single Executor with FIFO behavior. If an application creates Virtual Threads (hey, why not?! they are cheaper!) then they will be scheduled alongside the request-processing threads as well. Do I understand things correctly? Is my scenario of request-starvation for a little-used but potentially critical application (e.g. Manager) a real concern, or am I missing something fundamental about the way Virtual Threads are scheduled relative to other Virtual Threads, and relative to non-virtual thre
Re: CIS Tomcat 8 Benchmark (v1.1.0) -- Questions
Mark, Robert, et al, On 9/5/23 11:26, Mark Thomas wrote: I spoke to some CIS representatives at a recent conference given that I have concerns about the quality of some of the recommendations. It appears that these benchmarks are effectively crowdsourced. My primary concern is that there is no validation of the resulting recommendations so they run the risk of reinforcing current widespread bad practice as well as good practice. This is interesting information. Lore really isn't a good source of security information. :( It makes me wonder why they are charging for that kind of information if it's basically the output of an internet search for "how do i secure a Tomcat installation". Regarding the suggestion to edit the contents of a JAR, that is a bad idea on so many levels. +1 Assuming that you are using signed JAR files, what you end up with is a JAR whose signature can no longer be validated. This would require that you disable JAR signing, which is probably another no-no when it comes to security -- and I'd agree -- if you have signed JARs you ought to be validating them. Another option would be to re-sign the modified JAR file with an internal certificate but that can present its own challenges. Most dev teams would just implement some kind of lazy auto-signing process with the cert and key sitting right on the server where it's being used, and at that point you are giving an attacker everything they need to work-around your security controls. I'd argue that rather than spending time hiding the current Tomcat version - which is nothing more than security by obscurity - system admins should be investing time in an update process that allows easy updates (and roll-backs) to the Tomcat version. You only need to hide the version number if you aren't on top of your security updates. And if you aren't on top of your security updates you have bigger problems than needing to hide the version number. +1 https://tomcat.apache.org/presentations.html#latest-split-installation If you find you need to hide this version number to appease auditors - and using smarter auditors isn't an option - then there are a range of options as set out in the Tomcat 8.5 security guide. That guide also provides the correct way to override the version number (if you really need to) without editing the JAR contents. In short, you can simply override the individual file by placing at the right place in the file system: $CATALINA_BASE/lib/org/apache/catalina/util/ServerInfo.properties +1 The only argument /against/ doing that in the past that I've ever seen was that they "didn't want to pollute [their] Tomcat installation" which is crazy when the alternative is to alter a signed JAR provided by a trusted vendor. This is also probably worth a read: https://cwiki.apache.org/confluence/display/TOMCAT/Community+Review+of+DISA+STIG I would bet that many of the items listed here have overlap with some of the more dubious items coming from CIS. Hope that helps, -chris On 05/09/2023 14:54, Robert Turner wrote: Thanks Peter. Just to be clear that I have no concern about the goal of the test -- only the method for obtaining the information, and the "implied" correction. After going through the rest of the document I was provided, it seems to "get more modern" as the questions go on -- suggesting the method of improvement is additive, and possibly not corrective. Improvements are definitely corrective as well as additive. Early versions of the guide had very odd advice regarding MIME type mapping that has since been removed. On Tue, Sep 5, 2023 at 9:36 AM Peter Kreuser wrote: Robert, While Mark Thomas will have a more detailled answer to this... The finding behind this test is valid (information disclosure with server version in responses), though the remediation listed here is from looong time ago, when the was no ErrorReportValve to purge the version info. So the CIS Tomcat 8(!) Guide is pretty outdated! Probably in more than this spot... Peter Am 05.09.2023 um 14:03 schrieb Robert Turner : While I think I know the answer to my question, I wanted to double-check with the group to confirm. I have been asked to perform the CIS Apache Tomcat 8 Benchmark (v1.1.0) on our production Tomcat installation, and I am looking through the questions / information extraction requests, and I suspect they are not really evaluating what they think they are, and furthermore encouraging bad practices. For instance, the first entry I have in the spreadsheet I was provided is listed as follows: CIS Control: 2.1 Alter the Advertised server.info String (Scored) Description: The server.info attribute contains the name of the application service. This value is presented to Tomcat clients when clients connect to the tomcat server. Audit Procedures: Perform the following to determine if the server.info value has been changed: Extract the ServerInfo.properties file and exami
Virtual Threads
All, I have some questions about Virtual Threads and their use within Tomcat. Note that only Tomcat 11 currently has support for Virtual Threads when running on a version 19 or later JVM. My (admittedly limited) understanding is that the use of Virtual Threads, specifically within Tomcat, will allow Tomcat to use fewer platform threads (e.g. native OS threads) to support a larger number of JVM/application threads for request-processing. The justification for this improvement is that request-processing threads spend a lot of time in an I/O-blocked state which, when using Virtual Threads, would no longer consume a platform thread. There appears to only be one setting to enable Virtual Threads in Tomcat: or In both cases, there is only one setting to affect the number of "threads" (by any description) which the Executor will ultimately use (Connectors without an explicit Executor will create a non-shared Executor to be used internally). That setting is "maxThreads" which limits the total number of "threads" that the Executor will create. The implementation of the VirtualThreadExecutor does not seem to have any upper-bound on the number of Virtual Threads that will be created at all. I believe this means that a large number of incoming requests (i.e. a burst) will result in the creation of a large number of Virtual Threads. Without an upper-bound, we are expecting that the JVM's (virtual) thread scheduler will schedule each application thread to be mounted to a platform thread in the order it was added to the queue (essentially in request-order). Before Virtual Threads were introduced, the Executor would use a queue of requests to dispatch to available (non-Virtual) threads which would use that thread until the request was complete, then return the thread to the pool. With Virtual Threads, the same thing is essentially still happening except that (a) Tomcat no longer manages the thread pool (a JVM-defined one is being used instead) and (b) requests are immediately handed a (Virtual) thread to carry their execution. I believe there are some subtle differences in how Tomcat will behave, now. As an example, if I have two applications running, say, the Tomcat Manager application and the Tomcat Examples application, without using Virtual Threads, each application's thread pools should be "fair" within the context of each application: requests are processed in the order they are received in Manager and Examples, separately. If all requests are equally "expensive" and everything is "fair", then requests to the Examples application are scheduled alongside those to the Manager application, and they can both execute simultaneously as well as separately-manage the order in which the requests are processed. Once Virtual Threads are introduced, the requests are filed into a single JVM-wide thread-scheduling system where activity in one application can affect the other. Let's replace Examples with RealWorldApp, an application that is expected to be used by users. Maybe a LOT of users. Without Virtual Threads, a high number of requests to RealWorldApp will not cause starvation of requests to (maybe the more important, at least for admins) the Manager. Once Virtual Threads are introduced, a limitless number of requests can be queued in front of a request to Manager, which can experience starvation. While Tomcat did not previously implement any specific priority-queuing of requests, the use of separate Executors for each application effectively provided that kind of thing. Yes, each Executor can be configured separately to either use Virtual Threads or not, and so Manager can be configured to NOT use Virtual Threads while RealWorldApp can be configured to use Virtual Threads and the balance is "restored". But it is no longer possible to have RealWorldApp and RealWorldApp2 and Manager all with equally probable request-scheduling when using Virtual Threads. You can pick some subset of applications to get (essentially) priority by NOT using Virtual Threads, but the whole set of applications running in a single JVM will share a single Executor with FIFO behavior. If an application creates Virtual Threads (hey, why not?! they are cheaper!) then they will be scheduled alongside the request-processing threads as well. Do I understand things correctly? Is my scenario of request-starvation for a little-used but potentially critical application (e.g. Manager) a real concern, or am I missing something fundamental about the way Virtual Threads are scheduled relative to other Virtual Threads, and relative to non-virtual threads? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CIS Tomcat 8 Benchmark (v1.1.0) -- Questions
Thanks Mark -- I appreciate your insights, and I agree with not donating your time so someone else can make money off it. It definitely seems somewhat unethical on their part not to make those resources available for free if they are sourcing them for free. I would imagine it's still easy enough to build up a commercial service around it, even if the artifacts are all public. I have traditionally relied on the advice in the Tomcat documentation in precedence over anything the auditors "recommend" for the reasons you articulated. Unfortunately, "audit" has a "special place" in the world ... We do already override the "Server" header which is the obvious visible version information (as so many "audits" insist on this being a "good thing"), but I agree about staying on top of versions as the more correct solution (rant: if only AWS would keep their Elastic Beanstalk platform up to date as well...never mind that they skipped version 9, making it quite awkward to jump from 8.5 to 10). Thanks for the additional details on that one should it come up as a "finding". I was only using that one as a "example", more to start a discussion, and to ensure I wasn't "out to lunch" on my thoughts. There are plenty of others entries in the evaluation document where the method for determining is too narrow, and relies on a specific version (often old) or a particular location for configuration (like explicitly specifying options in catalina.sh, even though there are more appropriate places, like in $CATALINA_HOME/conf files, or in the applications), as well as some items that are somewhat obscure and don't result in improving security in any material way (i.e. obscuring, rather than securing). Robert On Tue, Sep 5, 2023 at 11:27 AM Mark Thomas wrote: > I spoke to some CIS representatives at a recent conference given that I > have concerns about the quality of some of the recommendations. > > It appears that these benchmarks are effectively crowdsourced. My > primary concern is that there is no validation of the resulting > recommendations so they run the risk of reinforcing current widespread > bad practice as well as good practice. > > The only response CIS had was to join the CIS community and try and > improve these guides. That is on my TODO list but it pains me to do so > as they charge for commercial use. I don't like the idea of providing my > expertise for free so someone else can make money out of it. I think I'd > rather spend time on the security guides in the Tomcat docs. > > Hmm. Looking back at my notes it appears I looked into doing something > similar back in 2010 and had similar concerns regarding the > participation terms back then too. > > Regarding the suggestion to edit the contents of a JAR, that is a bad > idea on so many levels. > > I'd argue that rather than spending time hiding the current Tomcat > version - which is nothing more than security by obscurity - system > admins should be investing time in an update process that allows easy > updates (and roll-backs) to the Tomcat version. You only need to hide > the version number if you aren't on top of your security updates. And if > you aren't on top of your security updates you have bigger problems than > needing to hide the version number. > > If you find you need to hide this version number to appease auditors - > and using smarter auditors isn't an option - then there are a range of > options as set out in the Tomcat 8.5 security guide. That guide also > provides the correct way to override the version number (if you really > need to) without editing the JAR contents. In short, you can simply > override the individual file by placing at the right place in the file > system: > > $CATALINA_BASE/lib/org/apache/catalina/util/ServerInfo.properties > > HTH, > > Mark > > > On 05/09/2023 14:54, Robert Turner wrote: > > Thanks Peter. Just to be clear that I have no concern about the goal of > the > > test -- only the method for obtaining the information, and the "implied" > > correction. > > > > After going through the rest of the document I was provided, it seems to > > "get more modern" as the questions go on -- suggesting the method of > > improvement is additive, and possibly not corrective. > > Improvements are definitely corrective as well as additive. Early > versions of the guide had very odd advice regarding MIME type mapping > that has since been removed. > > > > > > On Tue, Sep 5, 2023 at 9:36 AM Peter Kreuser wrote: > > > >> Robert, > >> > >> While Mark Thomas will have a more detailled answer to this... > >> > >> The finding behind this test is valid (information disclosure with > server > >> version in responses), though the remediation listed here is from looong > >> time ago, when the was no ErrorReportValve to purge the version info. > >> > >> So the CIS Tomcat 8(!) Guide is pretty outdated! Probably in more than > >> this spot... > >> > >> Peter > >> > >>> Am 05.09.2023 um 14:03 schrieb Robert Turner : > >>> > >>> While I think
Re: [External] Re: Supporting Proxy Protocol in Tomcat
Jon, On 9/4/23 10:41, Jonathan S. Fisher wrote: Mark thank you again for your leadership and setting expectations. I'm going to commit to working on this with anyone else that wants to help with the goal of a patch by year end. I want to nail the patch with minimal rework that meets Tomcat project quality standards. To that end, I'll attempt to summarize what you expect here and if you could comment and correct my understanding that would be appreciated. It sounds like you're satisfied with the ubiquity of the Proxy protocol and that it has an RFC +1 We'll target just implementing the latest version of the Proxy protocol We'll implement a "TrustedProxies" feature similar to what the Remote IP Valve does +1 This is very important. If it makes sense to do some refactoring to allow these features to share code, please do that as a completely separate commit in your PR. It will make things much easier to read and follow. We'll implement a, or modify the RemoteIp, valve to be able to set the remote IP from Proxy protocol headers We'll follow the RFC spec and reject any request that does a proper Proxy protocol header I'm particularly interested in the Proxy protocol over Unix Domain Sockets, so expect to see a lot of the work focused on this, but accepting Proxy Protocol over TCP looks to be quite important from the comments on this email chain TCP is a much more important use-case IMHO. If I may ask two things: Can you summarize your desired implementation? What point in the stack should we target to implement this? One thing I'm not familiar with on Tomcat is the testing expectations. If you can point to a set of unit tests and a set of integration tests and say "Do it like this" There are tons of unit tests in the Tomcat source tree. IMHO it's much more important to have /existing/ unit-tests than waiting for a long time to have "the perfect tests case(s)". If you commit decent unit-tests and there are opportunities to make them more "Tomcat-like" or share more code with existing unit-tests, that work can come later. Anything else on the original patch you liked/didn't like? ( https://bz.apache.org/bugzilla/show_bug.cgi?id=57830) Others know much more about the "right" way to get started on this, so I'll leave it to them to reply. I wouldn't want to lead you down the wrong path. -chris On Tue, Aug 29, 2023 at 3:13 PM Mark Thomas wrote: On 28/08/2023 18:44, Amit Pande wrote: Oh, sure. So, what would be the best way to get some conclusion on this thread? Provide a patch for review based on the feedback provided here and in the BZ issue. https://bz.apache.org/bugzilla/show_bug.cgi?id=57830 The state of the ticket isn't updated for long. Perhaps add comments/ask the folks on user list to vote? That is more likely to irritate folks rather than encourage them to help you progress your patch. Mark Thanks, Amit -Original Message- From: Mark Thomas Sent: Monday, August 28, 2023 11:20 AM To: Tomcat Users List Subject: Re: [External] Re: Supporting Proxy Protocol in Tomcat CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you believe this is a phishing email, use the Report to Cybersecurity icon in Outlook. 28 Aug 2023 17:11:20 Amit Pande : Mark, Just checking - Did this issue get discussed in any of the core members' meeting? There are no such meetings. Discussion happens on the mailing lists. Mark Thanks, Amit -Original Message- From: Amit Pande Sent: Monday, July 31, 2023 9:29 AM To: Tomcat Users List Subject: RE: [External] Re: Supporting Proxy Protocol in Tomcat Yes, understood. Thank you for clarifying. Even I was referring to initial consensus without any timeline or approach conclusion. Thanks, Amit -Original Message- From: Mark Thomas Sent: Friday, July 28, 2023 2:48 PM To: users@tomcat.apache.org Subject: Re: [External] Re: Supporting Proxy Protocol in Tomcat On 28/07/2023 19:21, Amit Pande wrote: Thank you all for the valuable discussion on this topic. Is it okay to say that we're agreeing to adding proxy protocol support in Tomcat? I think that is a little too strong. At this point there is a proposed approach and no one is objecting but until there is an actual patch to discuss... Keep in mind that any committer can veto a change. My sense is that it should be possible to implement this feature while addressing any concerns that may be raised but it is not guaranteed. Mark Thanks, Amit -Original Message- From: Christopher Schultz Sent: Thursday, July 27, 2023 4:13 PM To: users@tomcat.apache.org Subject: Re: [External] Re: Supporting Proxy Protocol in Tomcat All, On 7/27/23 12:39, Mark Thomas wrote: On 27/07/2023 16:27, Jonathan S. Fisher wrote: On the topic of security, may we consider a trustedProxies setting? Seems reasonable. We should probably look at what h
RE: [External] Re: Supporting Proxy Protocol in Tomcat
Thank you, Jonathan, for updating this thread. I was thinking of cleaning up the existing patch and submit it for review. I agree it's better to get feedback before any work is started. Thanks, Amit -Original Message- From: Jonathan S. Fisher Sent: Monday, September 4, 2023 9:42 AM To: Tomcat Users List Subject: Re: [External] Re: Supporting Proxy Protocol in Tomcat CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. If you believe this is a phishing email, use the Report to Cybersecurity icon in Outlook. Mark thank you again for your leadership and setting expectations. I'm going to commit to working on this with anyone else that wants to help with the goal of a patch by year end. I want to nail the patch with minimal rework that meets Tomcat project quality standards. To that end, I'll attempt to summarize what you expect here and if you could comment and correct my understanding that would be appreciated. It sounds like you're satisfied with the ubiquity of the Proxy protocol and that it has an RFC We'll target just implementing the latest version of the Proxy protocol We'll implement a "TrustedProxies" feature similar to what the Remote IP Valve does We'll implement a, or modify the RemoteIp, valve to be able to set the remote IP from Proxy protocol headers We'll follow the RFC spec and reject any request that does a proper Proxy protocol header I'm particularly interested in the Proxy protocol over Unix Domain Sockets, so expect to see a lot of the work focused on this, but accepting Proxy Protocol over TCP looks to be quite important from the comments on this email chain If I may ask two things: Can you summarize your desired implementation? What point in the stack should we target to implement this? One thing I'm not familiar with on Tomcat is the testing expectations. If you can point to a set of unit tests and a set of integration tests and say "Do it like this" Anything else on the original patch you liked/didn't like? ( https://bz.apache.org/bugzilla/show_bug.cgi?id=57830) Thank you, On Tue, Aug 29, 2023 at 3:13 PM Mark Thomas wrote: > On 28/08/2023 18:44, Amit Pande wrote: > > Oh, sure. So, what would be the best way to get some conclusion on > > this > thread? > > Provide a patch for review based on the feedback provided here and in > the BZ issue. > > > https://bz/ > > .apache.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D57830&data=05%7C01%7CAm > > it.Pande%40veritas.com%7Cead161f45fdf4f7ae40808dbad552039%7Cfc8e13c0 > > 422c4c55b3eaca318e6cac32%7C0%7C0%7C638294353363138987%7CUnknown%7CTW > > FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVC > > I6Mn0%3D%7C3000%7C%7C%7C&sdata=UPENCoX0wXwqKw4L96w9PfNxSvh%2FzItbDGg > > uwsdMlqY%3D&reserved=0 The state of the > ticket isn't updated for long. Perhaps add comments/ask the folks on > user list to vote? > > That is more likely to irritate folks rather than encourage them to > help you progress your patch. > > Mark > > > > > > Thanks, > > Amit > > > > -Original Message- > > From: Mark Thomas > > Sent: Monday, August 28, 2023 11:20 AM > > To: Tomcat Users List > > Subject: Re: [External] Re: Supporting Proxy Protocol in Tomcat > > > > > > CAUTION: This email originated from outside the organization. Do not > click links or open attachments unless you recognize the sender and > know the content is safe. If you believe this is a phishing email, use > the Report to Cybersecurity icon in Outlook. > > > > > > > > 28 Aug 2023 17:11:20 Amit Pande : > > > >> Mark, > >> > >> Just checking - Did this issue get discussed in any of the core > >> members' meeting? > > > > There are no such meetings. Discussion happens on the mailing lists. > > > > Mark > > > > > >> > >> Thanks, > >> Amit > >> > >> -Original Message- > >> From: Amit Pande > >> Sent: Monday, July 31, 2023 9:29 AM > >> To: Tomcat Users List > >> Subject: RE: [External] Re: Supporting Proxy Protocol in Tomcat > >> > >> Yes, understood. > >> > >> Thank you for clarifying. Even I was referring to initial consensus > >> without any timeline or approach conclusion. > >> > >> Thanks, > >> Amit > >> > >> -Original Message- > >> From: Mark Thomas > >> Sent: Friday, July 28, 2023 2:48 PM > >> To: users@tomcat.apache.org > >> Subject: Re: [External] Re: Supporting Proxy Protocol in Tomcat > >> > >> On 28/07/2023 19:21, Amit Pande wrote: > >>> Thank you all for the valuable discussion on this topic. > >>> > >>> Is it okay to say that we're agreeing to adding proxy protocol > >>> support in Tomcat? > >> > >> I think that is a little too strong. At this point there is a > >> proposed approach and no one is objecting but until there is an > >> actual patch to discuss... > >> > >> Keep in mind that any committer can veto a change. > >> > >> My sense is that it should be possible to implement this feature > >> while addressin
Re: CIS Tomcat 8 Benchmark (v1.1.0) -- Questions
I spoke to some CIS representatives at a recent conference given that I have concerns about the quality of some of the recommendations. It appears that these benchmarks are effectively crowdsourced. My primary concern is that there is no validation of the resulting recommendations so they run the risk of reinforcing current widespread bad practice as well as good practice. The only response CIS had was to join the CIS community and try and improve these guides. That is on my TODO list but it pains me to do so as they charge for commercial use. I don't like the idea of providing my expertise for free so someone else can make money out of it. I think I'd rather spend time on the security guides in the Tomcat docs. Hmm. Looking back at my notes it appears I looked into doing something similar back in 2010 and had similar concerns regarding the participation terms back then too. Regarding the suggestion to edit the contents of a JAR, that is a bad idea on so many levels. I'd argue that rather than spending time hiding the current Tomcat version - which is nothing more than security by obscurity - system admins should be investing time in an update process that allows easy updates (and roll-backs) to the Tomcat version. You only need to hide the version number if you aren't on top of your security updates. And if you aren't on top of your security updates you have bigger problems than needing to hide the version number. If you find you need to hide this version number to appease auditors - and using smarter auditors isn't an option - then there are a range of options as set out in the Tomcat 8.5 security guide. That guide also provides the correct way to override the version number (if you really need to) without editing the JAR contents. In short, you can simply override the individual file by placing at the right place in the file system: $CATALINA_BASE/lib/org/apache/catalina/util/ServerInfo.properties HTH, Mark On 05/09/2023 14:54, Robert Turner wrote: Thanks Peter. Just to be clear that I have no concern about the goal of the test -- only the method for obtaining the information, and the "implied" correction. After going through the rest of the document I was provided, it seems to "get more modern" as the questions go on -- suggesting the method of improvement is additive, and possibly not corrective. Improvements are definitely corrective as well as additive. Early versions of the guide had very odd advice regarding MIME type mapping that has since been removed. On Tue, Sep 5, 2023 at 9:36 AM Peter Kreuser wrote: Robert, While Mark Thomas will have a more detailled answer to this... The finding behind this test is valid (information disclosure with server version in responses), though the remediation listed here is from looong time ago, when the was no ErrorReportValve to purge the version info. So the CIS Tomcat 8(!) Guide is pretty outdated! Probably in more than this spot... Peter Am 05.09.2023 um 14:03 schrieb Robert Turner : While I think I know the answer to my question, I wanted to double-check with the group to confirm. I have been asked to perform the CIS Apache Tomcat 8 Benchmark (v1.1.0) on our production Tomcat installation, and I am looking through the questions / information extraction requests, and I suspect they are not really evaluating what they think they are, and furthermore encouraging bad practices. For instance, the first entry I have in the spreadsheet I was provided is listed as follows: CIS Control: 2.1 Alter the Advertised server.info String (Scored) Description: The server.info attribute contains the name of the application service. This value is presented to Tomcat clients when clients connect to the tomcat server. Audit Procedures: Perform the following to determine if the server.info value has been changed: Extract the ServerInfo.properties file and examine the server.info attribute. $ cd $CATALINA_HOME/lib $ jar xf catalina.jar org/apache/catalina/util/ServerInfo.properties $ grep server.info org/apache/catalina/util/ServerInfo.properties So, other than a few issues with the audit procedures, etc. This seems to be doing the following: a) evaluating a default value which I believe can be overridden and thus may not actually reflect the value the server may provide to external clients b) encouraging the modification of the catalina.jar contents to correct the default value There are a few similar items (for server.number, server.built) (2.2, 2.3). Thoughts / comments from "those in the know"? Thanks, Robert - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: CIS Tomcat 8 Benchmark (v1.1.0) -- Questions
Thanks Peter. Just to be clear that I have no concern about the goal of the test -- only the method for obtaining the information, and the "implied" correction. After going through the rest of the document I was provided, it seems to "get more modern" as the questions go on -- suggesting the method of improvement is additive, and possibly not corrective. On Tue, Sep 5, 2023 at 9:36 AM Peter Kreuser wrote: > Robert, > > While Mark Thomas will have a more detailled answer to this... > > The finding behind this test is valid (information disclosure with server > version in responses), though the remediation listed here is from looong > time ago, when the was no ErrorReportValve to purge the version info. > > So the CIS Tomcat 8(!) Guide is pretty outdated! Probably in more than > this spot... > > Peter > > > Am 05.09.2023 um 14:03 schrieb Robert Turner : > > > > While I think I know the answer to my question, I wanted to double-check > > with the group to confirm. > > > > I have been asked to perform the CIS Apache Tomcat 8 Benchmark (v1.1.0) > on > > our production Tomcat installation, and I am looking through the > questions > > / information extraction requests, and I suspect they are not really > > evaluating what they think they are, and furthermore encouraging bad > > practices. > > > > For instance, the first entry I have in the spreadsheet I was provided is > > listed as follows: > > > > CIS Control: > > 2.1 Alter the Advertised server.info String (Scored) > > > > Description: > > The server.info attribute contains the name of the application service. > > This value is presented to Tomcat clients when clients connect to the > > tomcat server. > > > > Audit Procedures: > > Perform the following to determine if the server.info value has been > > changed: > > Extract the ServerInfo.properties file and examine the server.info > > attribute. > > $ cd $CATALINA_HOME/lib > > $ jar xf catalina.jar org/apache/catalina/util/ServerInfo.properties > > $ grep server.info org/apache/catalina/util/ServerInfo.properties > > > > > > So, other than a few issues with the audit procedures, etc. This seems to > > be doing the following: > > > > a) evaluating a default value which I believe can be overridden and thus > > may not actually reflect the value the server may provide to external > > clients > > b) encouraging the modification of the catalina.jar contents to correct > the > > default value > > > > There are a few similar items (for server.number, server.built) (2.2, > 2.3). > > > > > > Thoughts / comments from "those in the know"? > > > > Thanks, > > > > Robert > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: CIS Tomcat 8 Benchmark (v1.1.0) -- Questions
Robert, While Mark Thomas will have a more detailled answer to this... The finding behind this test is valid (information disclosure with server version in responses), though the remediation listed here is from looong time ago, when the was no ErrorReportValve to purge the version info. So the CIS Tomcat 8(!) Guide is pretty outdated! Probably in more than this spot... Peter > Am 05.09.2023 um 14:03 schrieb Robert Turner : > > While I think I know the answer to my question, I wanted to double-check > with the group to confirm. > > I have been asked to perform the CIS Apache Tomcat 8 Benchmark (v1.1.0) on > our production Tomcat installation, and I am looking through the questions > / information extraction requests, and I suspect they are not really > evaluating what they think they are, and furthermore encouraging bad > practices. > > For instance, the first entry I have in the spreadsheet I was provided is > listed as follows: > > CIS Control: > 2.1 Alter the Advertised server.info String (Scored) > > Description: > The server.info attribute contains the name of the application service. > This value is presented to Tomcat clients when clients connect to the > tomcat server. > > Audit Procedures: > Perform the following to determine if the server.info value has been > changed: > Extract the ServerInfo.properties file and examine the server.info > attribute. > $ cd $CATALINA_HOME/lib > $ jar xf catalina.jar org/apache/catalina/util/ServerInfo.properties > $ grep server.info org/apache/catalina/util/ServerInfo.properties > > > So, other than a few issues with the audit procedures, etc. This seems to > be doing the following: > > a) evaluating a default value which I believe can be overridden and thus > may not actually reflect the value the server may provide to external > clients > b) encouraging the modification of the catalina.jar contents to correct the > default value > > There are a few similar items (for server.number, server.built) (2.2, 2.3). > > > Thoughts / comments from "those in the know"? > > Thanks, > > Robert - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
CIS Tomcat 8 Benchmark (v1.1.0) -- Questions
While I think I know the answer to my question, I wanted to double-check with the group to confirm. I have been asked to perform the CIS Apache Tomcat 8 Benchmark (v1.1.0) on our production Tomcat installation, and I am looking through the questions / information extraction requests, and I suspect they are not really evaluating what they think they are, and furthermore encouraging bad practices. For instance, the first entry I have in the spreadsheet I was provided is listed as follows: CIS Control: 2.1 Alter the Advertised server.info String (Scored) Description: The server.info attribute contains the name of the application service. This value is presented to Tomcat clients when clients connect to the tomcat server. Audit Procedures: Perform the following to determine if the server.info value has been changed: Extract the ServerInfo.properties file and examine the server.info attribute. $ cd $CATALINA_HOME/lib $ jar xf catalina.jar org/apache/catalina/util/ServerInfo.properties $ grep server.info org/apache/catalina/util/ServerInfo.properties So, other than a few issues with the audit procedures, etc. This seems to be doing the following: a) evaluating a default value which I believe can be overridden and thus may not actually reflect the value the server may provide to external clients b) encouraging the modification of the catalina.jar contents to correct the default value There are a few similar items (for server.number, server.built) (2.2, 2.3). Thoughts / comments from "those in the know"? Thanks, Robert
