Correction.
15. So I changed my code thus:
``extra_environ = {"HTTP_HOST": "localhost:443", "wsgi.url_scheme": "https"}
apptest.webtest.TestApp(app, extra_environ=extra_environ)''
On Wed, Oct 11, 2017 at 3:51 PM, Mike Orr <sluggos...@gmail.com> wrote:
> On Tue, Sep 26, 2017 at 8:14 AM, Michael Merickel <mmeri...@gmail.com> wrote:
>>> What is the Origin header and shouldn't Pyramid/WebOb set it automatically
>>> if it's becoming more important?
>>
>> 1) You can read the RFC about it or anything on google.
>> 2) The Origin header is set by the client, not the server.
>>
>> If the origin matches the current domain (usually set by the host header)
>> then the request is trusted by default and you do not need to modify
>> anything. You have your app setup in such a way that your requests *look*
>> like they are originating from another server instead of the domain hosting
>> the content. Just configure your webtest requests such that the origin and
>> host match and you'll be fine.
>>
>>> I still feel like there's a missing piece, something that needs to be
>>> documented so that others don't fall into this same trap.
>>
>> If you would like to contribute some documentation on this once you figure
>> it out I'm more than happy to review. It probably belongs in the testing
>> chapter about how to use webtest. If you think webtest should set the
>> origin/host the same by default then perhaps you could open an issue over
>> there.
>
> OK, I've got something that works; I'll make a cookbook page for
> testing with HTTPS-only cookies. I'll try to explain it precisely now
> although it's a long journey through webtest.app, webob.request,
> pyramid.csrf, and session cookies. To recap the situation:
>
> 1. The login form is CSRF-protected. (IT requirement.)
>
> 2. The app uses SessionAuthenticationPolicy and
> pyramid_redis_sessions. Cookies are HTTPS-only (IT requirement.), so
> setting 'redis.sessions.cookie_secure = true'. This tells the client
> to send the cookie only with HTTPS requests (not HTTP).
>
> 3. All this is working fine in production.
>
> 4. The functional test logs in via a form. It uses WebTest. There is
> no web server, just WebTest calling the WSGI app.
>
> 5. The test ran fine before implementing HTTPS cookies and CSRF protection.
>
> 6. HTTPS cookies broke the test, because WebTest emulates an HTTP
> environment by default, so it won't transmit the cookie that indicates
> the session containing the authenticated status. The login submit
> fails.
>
> 7.I made the request "HTTPS" with an environ var:
>
> ``extra_environ = {"wsgi.url_scheme": "https"}``.
> ``apptest = webtest.testApp(app, extra_environ=extra_environ}``.
> That fixed #6.
>
> 8. I implemented CSRF checking, and now the login failed because the
> "origin" (the client's domain) was different was different from what
> the "server" expected.
>
> 9. Troubleshooting and this list revealed two workarounds: turn off
> HTTPS-only cookies and revert to HTTP, or set the de facto host as a
> trusted host. (Setting 'pyramid.csrf_trusted_origins = localhost:80'.)
>
> 10. However, I didn't want to make my test configuration so different
> from the real one and lose some aspect of testing. Instead I wanted to
> understand the underlying problem and fix it.
>
> 11. The problem is that in the form submission request, the referer
> domain was different from the current domain. I don't have access to
> the request; it's internal to WebTest. I put print statements in
> 'pyramid.csrf' to determine what the request attributes were and what
> the mismatch was. The actual domain was 'localhost' while the referer
> had 'localhost:80'. It wasn't clear what code was calculating the
> referer value or what it based it on. It didn't make sense because
> didn't it know that 'localhost' is the same as 'localhost:80'?
>
> 12. Enlightenment came when I realized the referer was
> "https://localhost:80";, which is nonsense. The default HTTPS port is
> 443, not 80.
>
> 13. I tried setting extra_environ "HTTP_PORT" : "443" but that didn't
> work: the referer still had "https://localhost:80";.
>
> 14. I tracked down the code to the
> 'webob.request.BaseRequest.host_url' property. It looks first in
> envvar 'HTTP_HOST' and falls back to 'SERVER_NAME' and 'SERVER_PORT'.
>
> 15. So I changed my code thus:
>
> ``extra_environ = {"HTTP_HOST": "443", "wsgi.url_scheme": "https"}
> apptest.webtest.TestApp(app, extra_environ=extra_environ)''
>
> Presto, it works!
>
> I'm not sure if 'webtest' should be smarter, or if so exactly how. But
> at least this solves this underlying problem.
>
> Is there anything else I should do to more completely mimic an HTTPS request?
>
>
>
>> - Michael
>>
>>
>> On Tue, Sep 26, 2017 at 12:52 AM, Mike Orr <sluggos...@gmail.com> wrote:
>>>
>>> On Mon, Sep 25, 2017 at 10:40 PM, Mike Orr <sluggos...@gmail.com> wrote:
>>> > On Mon, Sep 25, 2017 at 9:00 PM, Mike Orr <sluggos...@gmail.com> wrote:
>>> >> On Mon, Sep 25, 2017 at 5:47 PM, Michael Merickel <mmeri...@gmail.com>
>>> >> wrote:
>>> >>>> So what's the best way forward?
>>> >>>
>>> >>> I think you covered your options pretty well.
>>> >>>
>>> >>> 1) Set wsgi.url_scheme to "http" as origin checks are only done on
>>> >>> https.
>>> >>> 2) Set the pyramid.csrf_trusted_origins as you are doing now.
>>> >>> 3) Disable csrf checking for your tests.
>>> >>>
>>> >>> I think it's just a helpful reminder that you would be wise to think
>>> >>> about
>>> >>> the origin header more these days as it's required by CORS requests
>>> >>> and, of
>>> >>> course, cross origin requests are the attack vector CSRF is helping to
>>> >>> protect.
>>> >>
>>> >> It sounds like it needs documentation then. What is the Origin header
>>> >> and shouldn't Pyramid/WebOb set it automatically if it's becoming more
>>> >> important?
>>> >>
>>> >> #1 and #3 would make the test environment different from the real
>>> >> environment. #2 raises the question of what is WebTest's Origin
>>> >> header, what should it be, why are they different, and does something
>>> >> need to be changed in the library?
>>> >
>>> > I guess the solution is #1, to roll back the HTTPS, because there is
>>> > no HTTPS because there's no network server. That in turn will require
>>> > a configuration that doesn't make the cookies HTTPS-only.
>>>
>>> (The browser got in a mood and did "Send" too quickly.)
>>>
>>> I still feel like there's a missing piece, something that needs to be
>>> documented so that others don't fall into this same trap. The only
>>> reason I set HTTPS-only and CSRF is our IT department asked us to do
>>> this wherever feasible, and since it didn't make much difference
>>> either way I went along with it. So presumably other people in other
>>> organizations will be doing the same thing, and have the same thing
>>> happen in their tests.
>>>
>>>
>>> --
>>> Mike Orr <sluggos...@gmail.com>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "pylons-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to pylons-discuss+unsubscr...@googlegroups.com.
>>> To post to this group, send email to pylons-discuss@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3Duo%2BAYpm6-oS-vPLFg0Ek3gz1G2JDHcW2m0MmdH_X8zp4g%40mail.gmail.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "pylons-discuss" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to pylons-discuss+unsubscr...@googlegroups.com.
>> To post to this group, send email to pylons-discuss@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/pylons-discuss/CAKdhhwFWJkrGFxOTkAyMDB61JkCm0%3D%3DwWONQ5V9U60D%3DAqV%3DXQ%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Mike Orr <sluggos...@gmail.com>
--
Mike Orr <sluggos...@gmail.com>
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to pylons-discuss+unsubscr...@googlegroups.com.
To post to this group, send email to pylons-discuss@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/pylons-discuss/CAH9f%3Duq_g7H%2BW1U%3DFp75%2B1NNHjCzPxKhgomEpAFsf1_utcw-sg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
