Hi Rodrigo, No worries, happy to have been able to help.
It sounds like the performance discrepancies you are seeing are quite strange! The SLA situation sounds tricky too, as it sounds like your server is responding in a reasonable amount of time (tens to hundreds of milliseconds), but you don't have any control over the network used to access the app, and its latency. What kind of concurrency do you have for your app servers? The only explanation I can think of would be that slow requests are hitting the CDN, which doesn't have a cached copy of the assets, and the CDN then has to fetch them from your app server, then serve them to the clients. This could be slow if your app server only has one process or thread, and there are several assets to be fetched in serial. It could also be slow if returning assets to the CDN is slow for some reason, like if you were accidentally compiling them on demand, rather than precompiling them. Another possibility would be if requests were being queued up at the routing layer (if you're using Heroku or similar), and that's where the time was being spent (rather than within your app itself). If your assets are changing a lot, and the number of users accessing the app is quite small, adding a CDN could feasibly slow things down - if most requests to each CDN edge location aren't able to return a cached asset, and have to fetch it from your server, you could be adding an extra DNS lookup and SSL handshake, just to have the CDN retrieve it from your server anyway. It might be worth looking at the cache hit rate for your CDN to see how often you're having to populate the CDN. I'm clutching at straws here though and just throwing things out there! :) In answer to your questions about browser extensions, at least for Chrome they can request permission to read and modify data on all sites you visit. They can also restrict themselves to being able to do this on a particular subset of domains. There's some more info about that here: https://developer.chrome.com/extensions/permission_warnings#warnings. It would also be possible for an extension to be interfering with requests in a way that slowed them down. In theory, adblocking or anti-virus extensions could exhibit this kind of behaviour. I haven't experienced it or seen it myself though. Anyway, it sounds like a tough problem to crack - things don't seem to be behaving as they should be. Good luck! Nick On Saturday, July 23, 2016 at 12:25:47 PM UTC+10, Rodrigo Rosenfeld Rosas wrote: > > Hi Nick, thanks for your response. > > Indeed the point that this is allowed to browser extensions is already > enough reason to not rely only on the Referer or Origin headers. > > "it would be unwise to turn off CSRF protection" - I never suggested to > turn CSRF protection off but I guess you meant to turn off the CSRF token > and I agree with you :) > > Our SPA is a bit atypical. Most SPA will probably perform some XHR > requests just after being loaded and it might perform pretty well except > for the first uncached request. However, our application is often updated > and not much accessed (a few hundred views per week) since it has a very > specific target (mostly attorneys). That means many of the requests do not > benefit from caching (we have a vendors JS that rarely changes, but the > other JS bundle is 65.8 KB after minified and gzipped which will probably > change after each new deploy). So, we try to make it perform well even for > the first uncached access. > > The main document is currently 10.6KB gzipped (more than a half of it is > JS injected by NewRelic middleware, which I'll probably try to move > elsewhere at some point since it would also invalidate the cache besides > the overhead). Or maybe I'll simply disable RUM as I already created > another tool which is the one I actually use to measure client-side > performance. Or maybe the NewRelic middleware is behind the ETag one and it > wouldn't affect caching, I'd have to check this. > > The trick to speed up the first uncached request is to load all data that > would be fetched with XHR directly in the document. We've done that once > and the result is that the main document became too big and it wasn't > cacheable and it was slowing down the download of the main page. Most of > this script wouldn't change often but it would add time even to cached > requests. So, I extracted those scripts into 3 separate requests, 2 of them > are usually in cache and the other one is quite small. And I load them by > using async scripts in the header. This way, as soon as the header is read > by the browser it can start downloading the other scripts (which usually > happens over HTTP/2 for most of our clients) which means they are > downloaded in parallel with the main document. But one of the scripts has > sensible information (the user's e-mail mostly). I could get the e-mail > using a separate XHR request, but I'd prefer to keep it simple. > > So, how would I serve dynamic JS from the Rails app (since they depend on > the user, they don't go to the CDN either) through GET requests (since they > are sourced in script tags) without making them vulnerable to CSRF? I have > to include the CSRF token in the query string. But if it changes on every > request to the main page it means the ETag generated by the middleware > would be always different, making it uncacheable (from the ETag middleware). > > That's why I'm asking if using the raw CSRF token stored in the session > would be enough protection given that our document doesn't reflect user > input in the query string. This way, since my understanding is that it > changes to avoid BREACH attacks and since the application is not vulnerable > to it if it doesn't reflect user's input, then I would be able to generate > a document which could be cached to improve the performance of cacheable > requests. > > After seeing that a small data is sent in the main document you might > think I'm not going to improve much more by making it cacheable, since > mobile devices are not supported by our application and most people have > enough bandwidth to download them pretty fast. However, this books explains > well that the main problem for the initial load is actually latency: > > https://hpbn.co/building-blocks-of-tcp/ > > I can't make the initial request closer to the users by serving them over > CDN (maybe it could help in some cases if the CDN has a faster connection > to our server than the user) because it can't be cached since it contains a > CSRF token that depends relies on the user's session. This means a > possibility to reduce latency would be to make the download size really low > (after some round trips, a large bandwidth will make it fast after the > latency), reducing the amount of round trips. If the main page was > cacheable it would have a low latency. If I have to provide the CSRF token > in that request then it will never be cacheable for the first access after > a new session is created (our sessions timeout after 2 hours of inactivity, > so this is often the case). So, now that I'm thinking about it, I guess > that it won't help me much to use the raw token anyway. I guess I'll > experiment with making the initial request cacheable and do not provide the > CSRF token on it but requesting it from a XHR request from some script tag > in the header. In that case, it won't matter whether the CSRF token would > be masked or not, so I don't have to worry about this. > > In case you are curious, for the past week, the user who loaded our > application the fastest took 274ms to load it. For that request most things > were cached and authentication was not involved. Usually, the user would > access a link containing an encrypted token which our application uses to > sign in the user. This is fast but the redirect to the main page sometimes > take a very long time and I can't explain the reason. The fastest load time > including this authentication phase and the redirect took 672ms. Both users > from US. "That's excelent, why are you caring about performance? Are you > crazy?" Well, those were the fastest access. We signed an SLA (against my > will) to serve users in under 5s since the link is followed. So, I'm pretty > happy if I can serve all our clients in within 5s. But 10% of our clients > take longer than 5s. The worst case this week was some user from US who > took 77s to load the application. Another user took 43s, another 19s, 13s, > 11s, 9s and so on. For the first case, redirecting to the root path after > the authentication took 73s (20ms in the Rails app as measured by nginx). I > have no idea why this happens to some users and how to fix it, but let's > look at other cases. > > An user from India took 43s to load the application. 2s in the redirect > part and, surprise, 40s for loading the main document (they always take > around 20ms in the server side accordingly to NewRelic and by inspecting > the nginx logs manually as well). I have no idea on why it happened since > the other resources were not cached and loaded very quickly while being > bigger than the main document. > > There's an even stranger case in which an US user would take 25s to load > one of our JS bundle from CDN, 24s to load another one and 9s to load the > CSS bundle (they are loaded in parallel). In that case the client took 29s > to load the full application. In the same request the main document was > loaded in 66ms but the user scripts, loaded from the same app, took around > 2s each to load (also in parallel). This was on IE11. > > It's interesting that everyweek I get about the same distribution and > about 10% of our clients take more than 5s to load the page. 60% of them > load it in under 2s. > > Since you work with browser extensions, maybe you'd be able to answer two > questions I have about them. Are they able to read data from any domain > using the user's cookies and GET requests? In that case, those extensions > would be able to get the CSRF token anyway, right? The other questions > regards the possibility of extensions delaying requests on purpose (maybe > some antivirus extension or something like that). Would that be possible > for browser extensions? Maybe those 10% of the users could be using such > extensions? > > I'm asking that because sometimes I see a weird behavior. For example, in > a requests that took 10s to load the application, I noticed that all > resources declared in the header only started downloaded immediately after > the main document has been fully loaded and it took about 2.3s to load. > Just after that the other requests started. And this happened in Chrome. > This is not the regular behavior I see in other requests as they start > before the main document is fully loaded as designed. So, is it possible > that this would happen with this user because he would be using some > extension or antivirus solution what could prevent those requests from > happening before the main document download completed? > > Here's a screenshot of our custom tool: > > > > Green is the latency and the black bar is the download time. > > If some user ever complain they are taking over 5s to load the page, would > I be right in telling them that this could be caused by some browser > extension? At least in cases like this. I don't know how to better > investigate the long redirect time issue... But since I reduced the main > document page I was able to see much better performance results in our > tool. I guess that making it cacheable over separate user sessions could > improve it furthermore. > > Anyway, thank you a lot for your valuable insight regarding browser > extensions! They also forced me to think more about the problem which made > me realize that using a fixed CSRF token to help with caching wouldn't > really help much as I thought initially. When I'm back to performance > improvements work I'll try to change the CSRF approach by requesting it in > a XHR request rather than embedding it in the main document, so that the > document could be served by our CDN more effectively (since it currently > depends on the user cookies/session it wouldn't be effective if enabled). > > Again, thank you very much! Good night :) > > Em 22-07-2016 20:21, Nicholas Firth-McCoy escreveu: > > Hi Rodrigo, > > It sounds like you're fairly across this problem domain. I wouldn't > consider myself an expert at this stuff, but everything you've said matches > my understanding of when and why CSRF protection is required. > > It's worth noting that browser extensions, malicious or not, can modify > the Origin and Referer headers (and CORS related headers). We do this at > Paydirt to allow us to insert our time-tracker iframe into sites that > normally disallow that in their CSP header. > > I think probably there's enough that outside your control (browser > versions, Flash/Java vulnerabilities, browser extensions) that it would be > unwise to turn off CSRF protection and just use the Origin header. > > Would it be possible to cache the SPA entry point page, and return CSRF > tokens when the client makes a GET request for the data payload? Or can > they send a POST/PUT requests before they necessarily fetch data from the > server? > > Cheers, > Nick > > On Friday, July 22, 2016 at 12:18:42 PM UTC+10, Rodrigo Rosenfeld Rosas > wrote: >> >> Em 21-07-2016 13:54, am...@pepo.com escreveu: >> >> Hi Rodrigo, >> >> CSRF tokens would prevent any malicious POST or PUT requests which are >> not made by the user knowingly.CSRF doesnot work for get requests at >> all.You can turn off CSRF if there aren't any POST request or those request >> wont do any damage to your application or database. >> >> >> Yes, I understand, but that doesn't answer my 2 questions. To make it >> short, given the scenario I described, I'd like to know: >> >> 1 - Why wouldn't checking Origin and Referer headers be enough to prevent >> CSRF? >> 2 - If the page containing the CSRF token does not reflect user input, >> would sending the CSRF token unmodified from the session be enough >> protection? >> >> Thanks, >> Rodrigo. >> >> P.S.: Rails doesn't protect GET requests because such requests shouldn't >> perform any state change in the server side, not because they are not >> vulnerable to CSRF. But if for some reason someone decides to take such >> actions on GET requests, there's nothing preventing them to use CSRF tokens >> on such requests as well. But for some GET request at least CSRF must be >> disabled so that one would be able to get the token before issues the next >> requests... Also, even if Rails would check for CSRF in GET requests it >> wouldn't be able to know which ones should be protected automatically... >> >> >> >> On Thursday, July 21, 2016 at 9:22:06 PM UTC+5:30, Rodrigo Rosenfeld >> Rosas wrote: >>> >>> A few extra comments to better explain it. >>> >>> Em 21-07-2016 11:54, Rodrigo Rosenfeld Rosas escreveu: >>> > Hi, hope to find someone here with a good understanding of CSRF >>> > handling in Rails and willing to answer my questions :) >>> > >>> > I've read the security guide and couldn't find this kind of >>> > information so maybe this is something we could improve in the guides >>> > and I could send a PR once I confirm I understood the reasoning behind >>> > each implementation details. >>> > >>> > So, what are my problems with CSRF tokens? They add complexity, >>> > prevent caching and seem unnecessary for my application. I know Rails >>> > is not designed for serving only my application but if we could >>> > improve the security guides I would be able to opt out from having a >>> > token if I could be sure my application would remain safe. Or, if I >>> > should keep using CSRF to be safe, then it would help if Rails >>> > wouldn't mask the token generating a new one on each request because >>> > that would allow the application to provide better caching. >>> > >>> > So, while researching about CSRF it seems like enough protection to >>> > simply verify the ORIGIN or REFERER headers unless there's a security >>> > breach in the browser itself or in some browser plugin, like Flash or >>> > Java. Sometimes those headers may not be present for legit requests, >>> > but this does not seem to be the case with the main browsers if your >>> > application runs over HTTPS in a single domain. I mean, Rails won't >>> > try to protect against GET requests to HTML pages as it doesn't make >>> > sense anyway, so in that case it would be fine if those headers would >>> > be missing or coming from a separate domain. >>> > >>> > So, as far as I can tell, checking the headers would be enough if the >>> > following conditions can be met: >>> > >>> > - the application doesn't change any data on GET requests, following >>> > the web semantics; >>> > >>> > - it's served over HTTPS for all requests (so that a request wouldn't >>> > be made to HTTP in the same domain which could hide the referer in >>> > some browser implementations); >>> >>> Using HTTPS also means it wouldn't be possible for a proxy server to >>> change such headers. >>> >>> > >>> > - it forbids non GET requests where those headers are both missing or >>> > when they don't match a whitelist; >>> > >>> > - the application restricts access to supported browsers; >>> > >>> > - the supported browsers and their plugins do not allow changing those >>> > headers for cross-domain requests; >>> > >>> > >>> > I understand the last two conditions are out of our control but the >>> > company would be able to decide whether or not they are responsible >>> > for buggy browsers or plugins. For example, if such an exploit is >>> > possible, then maybe it would also be possible that it could be used >>> > to issue a regular GET request from the other domain and extract the >>> > token from it anyway, right? >>> > >>> > Now, even if the company decides that they want to add extra >>> > protection anyway to make it harder for such bugged browsers/plugins >>> > exploits to be successfully used, maybe it would be enough to provide >>> > the unmasked CSRF token in the original GET request. It used to be the >>> > case in Rails but it was changed to mitigate BREACH attacks. >>> > Accordingly to http://breachattack.com/ the following conditions must >>> > be met for the application to be vulnerable: >>> > >>> > "If you have an HTTP response body that meets all the following >>> > conditions, you might be vulnerable: >>> > >>> > 1 - Your page is served with HTTP compression enabled (GZIP / DEFLATE) >>> > >>> > 2 - Your page reflects user data via query string parameters, POST... >>> > >>> > 3 - Your application page serves PII, a CSRF token, sensitive data..." >>> > >>> > While 1 and 3 would hold true for our SPA, we serve a single page >>> > which I'd want to be cacheable with ETag. We don't reflect query >>> > string parameters in that page, so I guess we would be safe from the >>> > BREACH attack. In that case we would be able to use the unmasked CSRF >>> > token stored in session, while remaining safe, right? >>> > >>> > I know Rails will try to add default protection that is suited to most >>> > applications but it would be helpful if it could explain better the >>> > measures it takes against the attacks in the security guide as well as >>> > explaining how to allow safe caching by telling in which conditions >>> > the CSRF token wouldn't have to be masked. I wouldn't be bothering you >>> > if the token masking didn't come with some caveat, but we actually >>> > have a trade-off here, as masking the token could affect the >>> > client-side performance by preventing proper caching (or by forcing >>> > the client to perform another XHR request to get the masked token). >>> >>> Actually it would be possible to cache using other techniques like >>> if-modified-since, getting a hash of the content with the token >>> extracted and so on, but I'd prefer to keep it simple and just use the >>> regular ETag middleware. >>> >>> > I'd love to understand if there are any other reasons why the CSRF >>> > token is being masked as well as confirming it would be okay to not >>> > mask it provided the page doesn't reflect any user input. If you know >>> > those answers I'll be pretty thankful for your feedback. >>> > >>> > >>> > Thanks, >>> > >>> > Rodrigo. >>> > >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Ruby on Rails: Core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to rubyonrails-co...@googlegroups.com. >> To post to this group, send email to rubyonra...@googlegroups.com. >> Visit this group at https://groups.google.com/group/rubyonrails-core. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Ruby on Rails: Core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to rubyonrails-co...@googlegroups.com <javascript:>. > To post to this group, send email to rubyonra...@googlegroups.com > <javascript:>. > Visit this group at https://groups.google.com/group/rubyonrails-core. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "Ruby on Rails: Core" group. To unsubscribe from this group and stop receiving emails from it, send an email to rubyonrails-core+unsubscr...@googlegroups.com. To post to this group, send email to rubyonrails-core@googlegroups.com. Visit this group at https://groups.google.com/group/rubyonrails-core. For more options, visit https://groups.google.com/d/optout.
