Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Nils Breunese]

Christopher Schultz  wrote:

>> Well yeah, it’s not like Envoy is a super niche proxy. We also found
>> the exact same issue in two other proxies in our network by the way.
>> Any proxy that does not consider path parameters when doing
>> path-based access control will have this issue when combined with a
>> server that does support them.
> This statement can be generalized to the following:
> 
> "When HTTP proxies and origin-servers disagree about how to process
> requests (specifically their URLs), Bad Things can happen."
> 
> I would expect most proxies to behave the way that Envoy evidently does:
> pass-through the URL in the request to the origin server. It's kind of a
> requirement of the HTTP spec(s).

That’s also what I expected. The issue is not about the URL that is passed on, 
but the way path access rules are evaluated.

> On the other hand, if you are applying a security-constraint at the
> proxy layer, I would expect that the server would at least explain how
> path-parameters are handled/normalized/removed/resolved/etc. Either
> Envoy doesn't publish that information, or you didn't read that section
> of the manual.

We’ve been in contacted with the Envoy people as well. Currently Envoy has no 
concept of path parameters, which is why the combination of Envoy with path 
access rules and Tomcat is vulnerable. But it seems that based on this thread 
Envoy might start ignoring path parameters when doing path access checks.

> There may even be a setting like
> normalize-URLs-to-be-proxied-for-authorization-checks or something like
> that. Or maybe not.

They are already normalized for the check, but without considering path 
parameters, which makes the system vulnerable. The normalization is only for 
the access check. If deemed allowed the URL is passed on unmodified from the 
original request AFAIK.

> But again, if the proxy and origin disagree, you'd better know the
> details and plan for them.

So we’ve learned now, yes. 

Nils.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Nils Breunese]

Christopher Schultz  wrote:

> On 9/24/20 07:46, Nils Breunese wrote:
>> Mark Thomas  wrote:
>> 
>>> On 24/09/2020 11:02, Nils Breunese wrote:
>>> 
>>> 
>>> 
>>>> - Envoy allows the request based on the /v1/* rule, because it 
>>>> does not support path parameters, because they are not part of
>>>> any recent standard (RFC 2396 dropped them in 1998 [1])
>>> 
>>> Envoy does support path parameters and is correctly doing so in this
>>> case as an HTTP reverse proxy by passing them through to the back-end.
>>> From an HTTP perspective "/bar" and "/foo/..;a=1/bar" are completely
>>> different URIs. Envoy's behaviour as an HTTP reverse proxy is correct.
>> 
>> I wasn’t suggesting that Envoy should modify the URL that it passes 
>> to the backend, but when doing its path-based access checks, I think
>> it should ignore path parameters and normalize the path, otherwise it
>> is trivial to get around these access restrictions.
>> 
>> When only allowing /v1/* Envoy won’t allow /v1/../internal/secrets,
>> because it normalises that to /internal/secrets and then sees it
>> doesn’t mach the access rule, but it will allow
>> /v1/..;color=red/internal/secrets because there is nothing to
>> normalize as far as it’s concerned.
> That seems inconsistent to me. If you remove the path parameter, you get
> /v1/../internal/secrets which would be normalized to /internal/secrets
> and therefore not allowed.

I’m not sure what you think is inconsistent. That is indeed what would happen 
if Envoy would ignore the path parameter when doing the check, but it doesn’t 
do that, because it doesn’t understand path parameters.

> But as a proxy, it should be forwarding the URLs as-is and may have
> different normalization behavior in that case.

I guess we were indeed operating under the assumption that URLs are URLs, but 
apparently different systems deal with them differently, which can cause 
unexpected behaviour and even security vulnerabilities. This was news to us.

> Honestly, if you want Envoy to enforce this kind of protection, it is
> Envoy you will need to configure properly. For example, if
> path-parameters are not used in your environment, consider simply
> rejecting any request that contains one and then your security layer
> will work as (otherwise) expected.

This is indeed the workaround we decided to go with, but it feels weird that we 
had to do this. I expect there are many other insecure setups around because of 
the different ways URLs are handled by various systems.

If Envoy would ‘understand’ path parameters and that they should be considered 
insignificant when doing path-based access checks, we would have been fine. But 
it seems Envoy and Tomcat don’t agree on whether path parameters are a thing or 
not. We’ll also talk to the Envoy people again about this.

>> We currently have a policy to not secure the applications themselves
> 
> Wow.

Ok, a little nuance: when it comes to path-based access restrictions.

>> , but handle this via generic access controls that can be centrally
>> audited and configured independently of what language or server is
>> used to implement the application.
> Okay. It looks like those generic access controls need a significant
> upgrade, then.

Well yeah, it’s not like Envoy is a super niche proxy. We also found the exact 
same issue in two other proxies in our network by the way. Any proxy that does 
not consider path parameters when doing path-based access control will have 
this issue when combined with a server that does support them.

Nils.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Nils Breunese]

Mark Thomas  wrote:

> On 24/09/2020 11:02, Nils Breunese wrote:
> 
> 
> 
>> - Envoy allows the request based on the /v1/* rule, because it does not 
>> support path parameters, because they are not part of any recent standard 
>> (RFC 2396 dropped them in 1998 [1])
> 
> Envoy does support path parameters and is correctly doing so in this
> case as an HTTP reverse proxy by passing them through to the back-end.
> From an HTTP perspective "/bar" and "/foo/..;a=1/bar" are completely
> different URIs. Envoy's behaviour as an HTTP reverse proxy is correct.

I wasn’t suggesting that Envoy should modify the URL that it passes to the 
backend, but when doing its path-based access checks, I think it should ignore 
path parameters and normalize the path, otherwise it is trivial to get around 
these access restrictions.

When only allowing /v1/* Envoy won’t allow /v1/../internal/secrets, because it 
normalises that to /internal/secrets and then sees it doesn’t mach the access 
rule, but it will allow /v1/..;color=red/internal/secrets because there is 
nothing to normalize as far as it’s concerned.

> Clearly there is scope for more education and documentation on this
> issue. The (very) short version is when using a reverse proxy in front
> of Tomcat:
> 1. Ideally don't rely on the reverse proxy to limit access to resources
> on Tomcat. Secure Tomcat as if everything was accessible.

We currently have a policy to not secure the applications themselves, but 
handle this via generic access controls that can be centrally audited and 
configured independently of what language or server is used to implement the 
application.

> 2. If you can't do 1, use mod_jk or the ISAPI redirector as they should
> map URIs to proxy targets using the same mapping rules as Tomcat (and if
> they don't we'll almost certainly treat that as a security issue).

Istio on Kubernetes uses Envoy as a proxy. I don’t think Envoy provides 
something like this.

> 3. If you can't do 1 or 2 you can try and block potentially malicious
> URIs in the reverse proxy but this is hard (need to think about "/../"
> and "/./" sequences, path parameters, %nn encoding and combinations of
> all three.
> 
> If you get as far as option 3, you really need to go back and
> re-consider option 1 or 2 as the chances of getting 3 100% right are slim.

AFAIK Envoy does normalize path traversal sequences like /../, but path 
parameters can break these sequences, and Envoy currently doesn’t ignore those, 
like the /v1/..;color=red/internal/secrets example.

Nils.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Nils Breunese]

Martin Grigorov  wrote:

> Someone else had the same/similar problem and the conclusion was that
> according to the Servlet specification this is the proper way to process
> the request - the request url should be normalized. If you need to protect
> some paths then you should do whatever is necessary in your application.

We have hundreds of applications running on Tomcat and path-based access 
control is currently handled outside Tomcat by Istio’s RBAC in the cloud. It 
appears that this is not a great match then.

> Please use secur...@tomcat.apache.org for reporting (possible) security
> problems in the future! Thanks!

I’m sorry. I read https://tomcat.apache.org/security.html 
 and it explicitly mentions using that 
address only for undisclosed security vulnerabilities. Since this issue seems 
to have been mentioned on the web in various places before, I thought it was 
fine to discuss this in public.

Nils.

Re: Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Nils Breunese]

Julian Reschke  wrote:

> Am 24.09.2020 um 12:02 schrieb Nils Breunese:
>> Hello,
>> 
>> I recently learned that when a server that supports path parameters [0] — 
>> like Tomcat (I found Jetty also does) — is run behind a reverse proxy that 
>> does path-based access control checks and does not support path parameters, 
>> your combined setup could be vulnerable.
>> 
>> Consider this setup:
>> 
>> 1. A Tomcat application without access restrictions
>> 2. An reverse proxy that only allows requests to /v1/* on the Tomcat 
>> application. I’ve used Envoy on Kubernetes, configured via Istio’s 
>> Role-Based Access Control (RBAC), but also observed this issue with other 
>> proxies.
>> 
>> Now I send a request for /v1/..;color=red/internal/secret to the proxy.
>> 
>> - Envoy allows the request based on the /v1/* rule, because it does not 
>> support path parameters, because they are not part of any recent standard 
>> (RFC 2396 dropped them in 1998 [1])
> > ...
> 
> That is very misleading.
> 
> RFC 2396 *extended* path parameters to be applicable to each path
> segment, see
> <https://greenbytes.de/tech/webdav/rfc2396.html#rfc.section.3.3>.
> 
> RFC 3986 further generalized things (see last paragraph of
> <https://www.greenbytes.de/tech/webdav/rfc3986.html#rfc.section.3.3>).
> 
> They have not been "dropped".

Ok, interesting. On the Istio issue tracker [0] I got a response saying: “Path 
segments were part of RFC2396, but they are not part of RFC3986 which obsoleted 
it, so they are not part of any active standard for over 15 years.” [0]

If they are still part of current standards, then the argument could be made 
that any system that does path-based access checks should ignore path 
parameters when checking a path, right? Because otherwise it is trivial to get 
around such path-based access restrictions.

Nils.

[0] https://github.com/istio/istio/issues/27409#issuecomment-696784022
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat's support for path parameters can expose resources despite reverse proxy access restrictions

[Nils Breunese]

Hello,

I recently learned that when a server that supports path parameters [0] — like 
Tomcat (I found Jetty also does) — is run behind a reverse proxy that does 
path-based access control checks and does not support path parameters, your 
combined setup could be vulnerable.

Consider this setup:

1. A Tomcat application without access restrictions
2. An reverse proxy that only allows requests to /v1/* on the Tomcat 
application. I’ve used Envoy on Kubernetes, configured via Istio’s Role-Based 
Access Control (RBAC), but also observed this issue with other proxies.

Now I send a request for /v1/..;color=red/internal/secret to the proxy.

- Envoy allows the request based on the /v1/* rule, because it does not support 
path parameters, because they are not part of any recent standard (RFC 2396 
dropped them in 1998 [1])
- Tomcat parses ;color=red as a path parameter, resolves the rest of the path 
to /v1/../internal/secret and consequently serves /internal/secret
- I’m pretty sure that whoever configured that RBAC policy did not want this to 
be possible

Note that:
- The client can request any URL on the Tomcat server via this path traversal 
attack.
- This also works with ‘empty’ path parameters, e.g. /v1/..;/internal/secret
- The URL doesn’t necessarily need to contain ‘..’ for naughtiness to be 
possible. When access to /public/secret would be restricted by the proxy, but 
other /public/* paths would be allowed, then an attacker would still be able to 
get the contents of /public/secret by requesting /public;/secret

The fact that different servers handle paths with path parameters differently 
seems to be known [2], but especially setups in which reverse proxies don’t 
support path parameters, but the servers behind them do, seems risky.

Would it be reasonable to create a ticket with a feature request for a flag to 
disable path parameter support in Tomcat? Or an even more secure suggestion: 
make path parameter support an opt-in feature, because it was dropped from the 
URI standard in 1998 and can make setups like I described vulnerable?

Thanks, Nils.

[0] RFC 1738 https://tools.ietf.org/html/rfc1738 
 (1994) and RFC 1808 
https://tools.ietf.org/html/rfc1808  
(1995).
[1] RFC 2396 G.4. Modifications from RFC 1808 
https://tools.ietf.org/html/rfc2396  
(1998). Path parameters are also absent in RFC 3986 
https://tools.ietf.org/html/rfc3986  
(2005).
[2] 
https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf
 

 slides 41 and further