I think you nailed it. Now that you're setting the host header properly
everything looks like it's coming from the same domain and thus none of
your requests look like they're cross-origin triggering extra checks. This
also fixes some things in url general I'm sure because your tests were
probably were generating urls like https://localhost:80/foo instead of
https://localhost/foo since your scheme is https and you set the host
header to the standard https port.
- Michael
On Wed, Oct 11, 2017 at 5:53 PM, Mike Orr <sluggos...@gmail.com> wrote:
> Correction.
>
> 15. So I changed my code thus:
>
> ``extra_environ = {"HTTP_HOST": "localhost:443", "wsgi.url_scheme":
> "https"}
> apptest.webtest.TestApp(app, extra_environ=extra_environ)''
>
>
> On Wed, Oct 11, 2017 at 3:51 PM, Mike Orr <sluggos...@gmail.com> wrote:
> > On Tue, Sep 26, 2017 at 8:14 AM, Michael Merickel <mmeri...@gmail.com>
> wrote:
> >>> What is the Origin header and shouldn't Pyramid/WebOb set it
> automatically
> >>> if it's becoming more important?
> >>
> >> 1) You can read the RFC about it or anything on google.
> >> 2) The Origin header is set by the client, not the server.
> >>
> >> If the origin matches the current domain (usually set by the host
> header)
> >> then the request is trusted by default and you do not need to modify
> >> anything. You have your app setup in such a way that your requests
> *look*
> >> like they are originating from another server instead of the domain
> hosting
> >> the content. Just configure your webtest requests such that the origin
> and
> >> host match and you'll be fine.
> >>
> >>> I still feel like there's a missing piece, something that needs to be
> >>> documented so that others don't fall into this same trap.
> >>
> >> If you would like to contribute some documentation on this once you
> figure
> >> it out I'm more than happy to review. It probably belongs in the testing
> >> chapter about how to use webtest. If you think webtest should set the
> >> origin/host the same by default then perhaps you could open an issue
> over
> >> there.
> >
> > OK, I've got something that works; I'll make a cookbook page for
> > testing with HTTPS-only cookies. I'll try to explain it precisely now
> > although it's a long journey through webtest.app, webob.request,
> > pyramid.csrf, and session cookies. To recap the situation:
> >
> > 1. The login form is CSRF-protected. (IT requirement.)
> >
> > 2. The app uses SessionAuthenticationPolicy and
> > pyramid_redis_sessions. Cookies are HTTPS-only (IT requirement.), so
> > setting 'redis.sessions.cookie_secure = true'. This tells the client
> > to send the cookie only with HTTPS requests (not HTTP).
> >
> > 3. All this is working fine in production.
> >
> > 4. The functional test logs in via a form. It uses WebTest. There is
> > no web server, just WebTest calling the WSGI app.
> >
> > 5. The test ran fine before implementing HTTPS cookies and CSRF
> protection.
> >
> > 6. HTTPS cookies broke the test, because WebTest emulates an HTTP
> > environment by default, so it won't transmit the cookie that indicates
> > the session containing the authenticated status. The login submit
> > fails.
> >
> > 7.I made the request "HTTPS" with an environ var:
> >
> > ``extra_environ = {"wsgi.url_scheme": "https"}``.
> > ``apptest = webtest.testApp(app, extra_environ=extra_environ}``.
> > That fixed #6.
> >
> > 8. I implemented CSRF checking, and now the login failed because the
> > "origin" (the client's domain) was different was different from what
> > the "server" expected.
> >
> > 9. Troubleshooting and this list revealed two workarounds: turn off
> > HTTPS-only cookies and revert to HTTP, or set the de facto host as a
> > trusted host. (Setting 'pyramid.csrf_trusted_origins = localhost:80'.)
> >
> > 10. However, I didn't want to make my test configuration so different
> > from the real one and lose some aspect of testing. Instead I wanted to
> > understand the underlying problem and fix it.
> >
> > 11. The problem is that in the form submission request, the referer
> > domain was different from the current domain. I don't have access to
> > the request; it's internal to WebTest. I put print statements in
> > 'pyramid.csrf' to determine what the request attributes were and what
> > the mismatch was. The actual domain was 'localhost' while the referer
> > had 'localhost:80'. It wasn't clear what code was calculating the
> > referer value or what it based it on. It didn't make sense because
> > didn't it know that 'localhost' is the same as 'localhost:80'?
> >
> > 12. Enlightenment came when I realized the referer was
> > "https://localhost:80";, which is nonsense. The default HTTPS port is
> > 443, not 80.
> >
> > 13. I tried setting extra_environ "HTTP_PORT" : "443" but that didn't
> > work: the referer still had "https://localhost:80";.
> >
> > 14. I tracked down the code to the
> > 'webob.request.BaseRequest.host_url' property. It looks first in
> > envvar 'HTTP_HOST' and falls back to 'SERVER_NAME' and 'SERVER_PORT'.
> >
> > 15. So I changed my code thus:
> >
> > ``extra_environ = {"HTTP_HOST": "443", "wsgi.url_scheme": "https"}
> > apptest.webtest.TestApp(app, extra_environ=extra_environ)''
> >
> > Presto, it works!
> >
> > I'm not sure if 'webtest' should be smarter, or if so exactly how. But
> > at least this solves this underlying problem.
> >
> > Is there anything else I should do to more completely mimic an HTTPS
> request?
> >
> >
> >
> >> - Michael
> >>
> >>
> >> On Tue, Sep 26, 2017 at 12:52 AM, Mike Orr <sluggos...@gmail.com>
> wrote:
> >>>
> >>> On Mon, Sep 25, 2017 at 10:40 PM, Mike Orr <sluggos...@gmail.com>
> wrote:
> >>> > On Mon, Sep 25, 2017 at 9:00 PM, Mike Orr <sluggos...@gmail.com>
> wrote:
> >>> >> On Mon, Sep 25, 2017 at 5:47 PM, Michael Merickel <
> mmeri...@gmail.com>
> >>> >> wrote:
> >>> >>>> So what's the best way forward?
> >>> >>>
> >>> >>> I think you covered your options pretty well.
> >>> >>>
> >>> >>> 1) Set wsgi.url_scheme to "http" as origin checks are only done on
> >>> >>> https.
> >>> >>> 2) Set the pyramid.csrf_trusted_origins as you are doing now.
> >>> >>> 3) Disable csrf checking for your tests.
> >>> >>>
> >>> >>> I think it's just a helpful reminder that you would be wise to
> think
> >>> >>> about
> >>> >>> the origin header more these days as it's required by CORS requests
> >>> >>> and, of
> >>> >>> course, cross origin requests are the attack vector CSRF is
> helping to
> >>> >>> protect.
> >>> >>
> >>> >> It sounds like it needs documentation then. What is the Origin
> header
> >>> >> and shouldn't Pyramid/WebOb set it automatically if it's becoming
> more
> >>> >> important?
> >>> >>
> >>> >> #1 and #3 would make the test environment different from the real
> >>> >> environment. #2 raises the question of what is WebTest's Origin
> >>> >> header, what should it be, why are they different, and does
> something
> >>> >> need to be changed in the library?
> >>> >
> >>> > I guess the solution is #1, to roll back the HTTPS, because there is
> >>> > no HTTPS because there's no network server. That in turn will require
> >>> > a configuration that doesn't make the cookies HTTPS-only.
> >>>
> >>> (The browser got in a mood and did "Send" too quickly.)
> >>>
> >>> I still feel like there's a missing piece, something that needs to be
> >>> documented so that others don't fall into this same trap. The only
> >>> reason I set HTTPS-only and CSRF is our IT department asked us to do
> >>> this wherever feasible, and since it didn't make much difference
> >>> either way I went along with it. So presumably other people in other
> >>> organizations will be doing the same thing, and have the same thing
> >>> happen in their tests.
> >>>
> >>>
> >>> --
> >>> Mike Orr <sluggos...@gmail.com>
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> Groups
> >>> "pylons-discuss" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send
> an
> >>> email to pylons-discuss+unsubscr...@googlegroups.com.
> >>> To post to this group, send email to pylons-discuss@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/pylons-discuss/CAH9f%
> 3Duo%2BAYpm6-oS-vPLFg0Ek3gz1G2JDHcW2m0MmdH_X8zp4g%40mail.gmail.com.
> >>> For more options, visit https://groups.google.com/d/optout.
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "pylons-discuss" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> an
> >> email to pylons-discuss+unsubscr...@googlegroups.com.
> >> To post to this group, send email to pylons-discuss@googlegroups.com.
> >> To view this discussion on the web visit
> >> https://groups.google.com/d/msgid/pylons-discuss/
> CAKdhhwFWJkrGFxOTkAyMDB61JkCm0%3D%3DwWONQ5V9U60D%3DAqV%3DXQ%
> 40mail.gmail.com.
> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >
> >
> >
> > --
> > Mike Orr <sluggos...@gmail.com>
>
>
>
> --
> Mike Orr <sluggos...@gmail.com>
>
> --
> You received this message because you are subscribed to the Google Groups
> "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylons-discuss+unsubscr...@googlegroups.com.
> To post to this group, send email to pylons-discuss@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/pylons-discuss/CAH9f%3Duq_g7H%2BW1U%3DFp75%
> 2B1NNHjCzPxKhgomEpAFsf1_utcw-sg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwGk_mAQPMAr22xYBhmRDh3366xyhF6wEkcm0Ju4i9AH2g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
