Aw: Re: Re: Fix for CVE-2020-1938
Hi Mark, so I went with your proposed default setting of "::". Alas, this fails, e.g., when we put our Tomcats into Docker containers, where only IPv4 is available (I would expect the same to happen on AWS environments, which are usually IPv4 only, too, haven't checked that yet): 10-Mar-2020 04:07:58.435 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-0:0:0:0:0:0:0:0-8009"] 10-Mar-2020 04:07:58.449 SEVERE [main] org.apache.catalina.core.StandardService.initInternal Failed to initialize connector [Connector[AJP/1.3-8009]] org.apache.catalina.LifecycleException: Protocol handler initialization failed at org.apache.catalina.connector.Connector.initInternal(Connector.java:1032) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) [...] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:303) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473) Caused by: java.net.SocketException: Protocol family unavailable at java.base/sun.nio.ch.Net.bind0(Native Method) at java.base/sun.nio.ch.Net.bind(Unknown Source) at java.base/sun.nio.ch.Net.bind(Unknown Source) at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(Unknown Source) at java.base/sun.nio.ch.ServerSocketAdaptor.bind(Unknown Source) at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:221) at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1118) at org.apache.tomcat.util.net.AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:223) at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:587) at org.apache.catalina.connector.Connector.initInternal(Connector.java:1030) ... 13 more So this means that I need to set the value to "::" in dual stack or pure IPv6 environments, while it seems that I have to use 0.0.0.0 in IPv4-only environments. I would therefore need to put some possibly fragile logic into various places in my startscripts (we have start scripts actually using Ant in our custom deployment solution, and for Docker we use bash) that determines the available IP versions, since I want to have as little config hazzle as possible for customers. I would rather avoid that and have a way to get back the old behaviour. Some of our apps are spring-boot-based with an embedded Tomcat. Here, we can just set the address to null in the programmatic configuration of the AJP connector that we created (spring boot has no built-in AJP support) and we seem to have the old behavior back. What we would need for Tomcats that are configured in the conventional way (i.e., server.xml) is to have a string value for the address attribute that translates to a null value for the bind address. Setting the address attribute to the empty string gives me "loopback only" behaviour. How about "*" to get back the old behavior? Regards J > Gesendet: Donnerstag, 05. März 2020 um 09:34 Uhr > Von: "Mark Thomas" > An: users@tomcat.apache.org > Betreff: Re: Aw: Re: Fix for CVE-2020-1938 > On 05/03/2020 07:12, "Jürgen Göres" wrote: > > >>> My first question is: what value do I need to set in the "address" > >>> attribute to indicate that I want the connector to listen on ALL > >>> interfaces (for IPv4 AND IPv6)? Maybe that should be documented. :-) > >> > >> It will vary by system. Some systems can't listen on both IPv4 and IPv6 > >> with a single socker. Usually either "::" or "0.0.0.0" will have the > >> desired result. > > > > That is a bit of a problem for us. In the environments we support (Win and > > Linux), from my observation the connectors would successfully bind to both > > IPv4 and IPv6 addresses. Since we have customers that use either IPv4, > > IPv6, or both and often have multiple interfaces on their machines, we > > cannot know which address/interface (or even which IP version) to bind to. > > And up to now, we didn't have to worry about it. > > Now our installation routine would somehow need to find out whether it > > should put a "::" or a "0.0.0.0" in the "address" attribute. What was "zero > > conf" for us so far now suddently becomes a new source for problems > > (=customer calls). > > Is there no way to optionally configure the old binding behavio
Aw: Re: Fix for CVE-2020-1938
> >Ghostcat is the name of a malware strain that has been around since at >least October last year. When referencing vulnerabilities it is best to >stick to the CVE reference since they should be unique (and if something >goes wrong and they aren't there are procedures to get them re-issued so >they are). > >> we are using Tomcat 9.0.x and 8.5.x in our stack. We make use of the AJP >> protocol since we use Apache HTTPD as reverse proxy and found it to be >> mostly hazzle-free over the last few years, so we would like to continue >> using it. >> Since the HTTPD and the Tomcats are in general not on the same nodes, the >> AJP connector has to listen on all interfaces. >> My first question is: what value do I need to set in the "address" attribute >> to indicate that I want the connector to listen on ALL interfaces (for IPv4 >> AND IPv6)? Maybe that should be documented. :-) > >It will vary by system. Some systems can't listen on both IPv4 and IPv6 >with a single socker. Usually either "::" or "0.0.0.0" will have the >desired result. That is a bit of a problem for us. In the environments we support (Win and Linux), from my observation the connectors would successfully bind to both IPv4 and IPv6 addresses. Since we have customers that use either IPv4, IPv6, or both and often have multiple interfaces on their machines, we cannot know which address/interface (or even which IP version) to bind to. And up to now, we didn't have to worry about it. Now our installation routine would somehow need to find out whether it should put a "::" or a "0.0.0.0" in the "address" attribute. What was "zero conf" for us so far now suddently becomes a new source for problems (=customer calls). Is there no way to optionally configure the old binding behaviour for the AJP connector? >> So the question is: is the root cause actually fixed? > >Yes. Great, thx. :-) >The very nature of the AJP protocol is such that clients have to be trusted. >Tomcat trusts the "real" client IP address provided. That could be used >for access control. >Tomcat may be configured to trust the authenticated user name provided. >That almost certainly will then be used for access control. So far, we had instructions for our customers to not expose the AJP ports (or any other internal ports of other components in our stack). However, I am pretty sure that there are installations out there where these ports are exposed (hopefully only on the intranet). In any case, we will update to the latest Tomcat version (but need to find a way to solve the "to which interface should we bind" problem without too much hazzle for customers), and in addition will also try to use the "secret" approach to secure access to the AJP connectors, so that even those customers that ignored the security guidelines will not be affected by other possible attacks over the AJP protocol. > >> If it is, what is the recommended mitigation? We consider using the "secret" >> feature (the filtering by request attributes is infeasible for us), but that >> would be a bit of effort and we are in a hurry. > >It would be unusual for an application to be using request attributes >directly. These have to be explicitly set in the reverse proxy and are >normally used for the reverse proxy to pass information to Tomcat about >the client etc. I was under the naive assumptions that "request attributes" meant "HTTP request attributes". ;-) If I understand you right here we are talking about attributes used in the AJP protocol to convey info between reverse proxy and Tomcat. Like info about the true client IP address (i.e., an AJP equivalent to HTTP's X-Forwarded-For header). >- setting up a dedicated subnet for reverse proxy to Tomcat >communication; >- configuring a firewall on the Tomcat box to only accept connections >to the AJP port from specific hosts That is what we advise our customers to do. >For any network configuration you can configure a shared secret for the >reverse proxy workers and the AJP connector. And since this is closest to "zero config" we can get, this is what we will do on perspective. Regards JG - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Aw: Re: Fix for the Ghostcat vulnerability
Hi, >> If it is, what is the recommended mitigation? We consider using the >> "secret" feature (the filtering by request attributes is infeasible >> for us), but that would be a bit of effort and we are in a hurry. >> > >We're in the same position as you. External web servers talking to >Tomcat servers on other boxes via AJP. > >We've looked at a few options, none of which seemed great: > >* The current stable version of Apache doesn't support the 'secret' >attribute for AJP connectors in mod_proxy. we will use the "secret" approach. Since we use mod_jk which supports it, this will offer the least trouble when deploying in customer environments. We will generate a random secret for each tomcat instance. Since our apps already register in our service registry,we can just add the secret there. Our Apache HTTPD resp. a little tooling we wrote for it that generates the Apache config from information in the registry and can pick up the secret from there as well. Regards JG - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Fix for the Ghostcat vulnerability
Hi, we are using Tomcat 9.0.x and 8.5.x in our stack. We make use of the AJP protocol since we use Apache HTTPD as reverse proxy and found it to be mostly hazzle-free over the last few years, so we would like to continue using it. Since the HTTPD and the Tomcats are in general not on the same nodes, the AJP connector has to listen on all interfaces. My first question is: what value do I need to set in the "address" attribute to indicate that I want the connector to listen on ALL interfaces (for IPv4 AND IPv6)? Maybe that should be documented. :-) It is clear that the AJP ports should not be exposed to the end users (let alone the public internet ;-)) and we of course make sure that this is the case for our SaaS hosting, but on-premise customers tend to ignore the security guidelines we provide with our product and might have AJP connectors not firewalled off. So it could be that there are installations out there which have the AJP ports exposed. This leads me to trhe next question is about the fixes themselves. I checked the changelists mentioned on these pages: https://tomcat.apache.org/security-8.html#Fixed_in_Apache_Tomcat_8.5.51 https://tomcat.apache.org/security-9.html#Fixed_in_Apache_Tomcat_9.0.31 >From what I gathered, these changes only disable the AJP connector by default, >change the listining interface to loopback (from "all interfaces"), add a >feature to filter requests by request attributes and change the way the >optional secret used between HTTPD and Tomcat is configured. I could not see any changes that actually resolve the issue in the AJP connector's implementation that allow the exploit. So the question is: is the root cause actually fixed? Or will an AJP connector that is (perhaps by accident) exposed still be vulnerable to the vulnerability? If it is, what is the recommended mitigation? We consider using the "secret" feature (the filtering by request attributes is infeasible for us), but that would be a bit of effort and we are in a hurry. Regards JG - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
