Re: [google-appengine] Re: Help resolve massive performance regression in 2.7 vs 2.5 runtime
Hi Pol, Thanks for getting back to me. On Sun, Nov 13, 2011 at 4:50 PM, Pol wrote: > Hi Brian, > > threadsafe is "true" There is a known issue where concurrent requests (enabled with threadsafe) can be much slower than non-concurrent requests, especially if the request is CPU-bound. You might want to set threadsafe to "false" and see if that fixes the problem. Cheers, Brian > Here's an example password_hash for an original 8 characters password: > > $2a$04$cbM2uHwDphIG3jFFRpq1mui5aVjevnDUwhvQ77S/WG/qvJMpiXAL6 > > On Nov 12, 6:58 pm, Brian Quinlan wrote: >> Hi Pol, >> >> On Sun, Nov 13, 2011 at 1:48 PM, Pol wrote: >> > Hi, >> >> > Since switching to 2.7 runtime, logging in tohttp://www.everpix.com >> > went from about a second to anywhere from 15s to 60s. I tracked it >> > down to this single password checking line: >> >> > from bcrypt import bcrypt >> > bcrypt.hashpw(password, self.password_hash) == self.password_hash >> >> What value are you using for "threadsafe" in your app.yaml? >> >> How large is self.password_hash? >> >> Cheers, >> Brian >> >> >> >> >> >> >> >> > This comes from "a native Python implementation of the py-bcrypt >> > package fromhttp://www.mindrot.org/projects/py-bcrypt/"; grabbed from >> > here:https://github.com/erlichmen/py-bcrypt. >> >> > So what's happening here and how can we fix this? >> >> > Thanks, >> >> > - Pol >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to google-appengine@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Help resolve massive performance regression in 2.7 vs 2.5 runtime
Hi Pol, Thanks for trying it again with threadsafe disabled and reporting back! On Mon, Nov 14, 2011 at 4:32 AM, Pol wrote: > Hi Brian, > > It helps, it's now at ~5s instead, but still at least 2x slower than > on the 2.5 runtime. > > So on December 1st, the 50% discount for front-instances is gone. The > idea is to compensate by switching to Python 2.7 with multithreading, > but it looks like at this point it's a lose-lose situation: it runs > more requests at the same time, but they take longer. We're mid- > November already, do you guys think you'll have all of this working > perfectly within 2 weeks? No, the issues with concurrent requests won't be fixed by the end of November. But note that concurrent requests will *not* improve the utilization of CPU-bound requests. Running multiple threads on the same CPU just proportionally slows each thread down. > The 1.6 SDK which actually allows to run 2.7 > locally was just released, so we're only starting to test now. Seems > to me deferring the payment change 1 more month to January 1st 2012 > wouldn't hurt :) You can send your billing-related suggestions to App Engine's Engineering Directory, Peter Magnusson (p...@google.com). Thanks! Cheers, Brian > On Nov 12, 10:09 pm, Brian Quinlan wrote: >> Hi Pol, >> >> Thanks for getting back to me. >> >> On Sun, Nov 13, 2011 at 4:50 PM, Pol wrote: >> > Hi Brian, >> >> > threadsafe is "true" >> >> There is a known issue where concurrent requests (enabled with >> threadsafe) can be much slower than non-concurrent requests, >> especially if the request is CPU-bound. >> >> You might want to set threadsafe to "false" and see if that fixes the >> problem. >> >> Cheers, >> Brian >> >> >> >> >> >> >> >> > Here's an example password_hash for an original 8 characters password: >> >> > $2a$04$cbM2uHwDphIG3jFFRpq1mui5aVjevnDUwhvQ77S/WG/qvJMpiXAL6 >> >> > On Nov 12, 6:58 pm, Brian Quinlan wrote: >> >> Hi Pol, >> >> >> On Sun, Nov 13, 2011 at 1:48 PM, Pol wrote: >> >> > Hi, >> >> >> > Since switching to 2.7 runtime, logging in tohttp://www.everpix.com >> >> > went from about a second to anywhere from 15s to 60s. I tracked it >> >> > down to this single password checking line: >> >> >> > from bcrypt import bcrypt >> >> > bcrypt.hashpw(password, self.password_hash) == self.password_hash >> >> >> What value are you using for "threadsafe" in your app.yaml? >> >> >> How large is self.password_hash? >> >> >> Cheers, >> >> Brian >> >> >> > This comes from "a native Python implementation of the py-bcrypt >> >> > package fromhttp://www.mindrot.org/projects/py-bcrypt/"; grabbed from >> >> > here:https://github.com/erlichmen/py-bcrypt. >> >> >> > So what's happening here and how can we fix this? >> >> >> > Thanks, >> >> >> > - Pol >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups "Google App Engine" group. >> >> > To post to this group, send email to google-appengine@googlegroups.com. >> >> > To unsubscribe from this group, send email to >> >> > google-appengine+unsubscr...@googlegroups.com. >> >> > For more options, visit this group >> >> > athttp://groups.google.com/group/google-appengine?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to google-appengine@googlegroups.com. >> > To unsubscribe from this group, send email to >> > google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Help resolve massive performance regression in 2.7 vs 2.5 runtime
Hi Pierre, On Tue, Nov 15, 2011 at 3:19 AM, Pol wrote: > Hi Brian, > >> > So on December 1st, the 50% discount for front-instances is gone. The >> > idea is to compensate by switching to Python 2.7 with multithreading, >> > but it looks like at this point it's a lose-lose situation: it runs >> > more requests at the same time, but they take longer. We're mid- >> > November already, do you guys think you'll have all of this working >> > perfectly within 2 weeks? >> >> No, the issues with concurrent requests won't be fixed by the end of >> November. >> >> But note that concurrent requests will *not* improve the utilization >> of CPU-bound requests. Running multiple threads on the same CPU just >> proportionally slows each thread down. > > That doesn't make sense: apps do a mix of CPU stuff and RPC stuff (and > possibly URL requests). What's the points of concurrent requests if it > slows down the CPU stuff while allowing to parallelize your RPC calls? This pattern (a mix of CPU use and RPC calls) will benefit from concurrent requests. I was writing about what I understood to be your login example. Presumably it does a single datastore read to access user information (taking 40ms of so) and then spends 1 seconds doing cryptography. > The end result will be the same number of instance as requests end up > taking longer. Isn't the scheduler supposed to watch all this and > ensure the CPU on each physical machine is not saturated? > > Only apps that do long poll URL requests and barely use the CPU would > benefit of concurrent requests then. > > We were told: don't worry so much about hours-based pricing, just wait > for 2.7 runtime, it'll have concurrent requests, it'll compensate > things. Clearly that doesn't work as promised if just turning > threadsafe ON makes a 2 seconds requests turn into a 30-60 seconds > one: the scheduler is not doing the right thing. Yes, these large latency increases are a bug: http://code.google.com/p/googleappengine/issues/detail?id=6323 > It seems what you need is a per WSGIApplication instance control of > the concurrent setting instead of global one, so you can turn it on > only where it makes sense. > > Finally, no matter what, concurrent or not, there's still a problem as > 2.7 runtime appears slower than 2.5 in this simple empirical test. I'm > starting to suspect you are using the 2.7 transition as a opportunity > to run more virtual instances per physical machine. That's not the case. The Python 2.7 runtime is slower than the Python 2.5 runtime in some cases and faster in others. We aren't publicizing the reasons why at this point. Cheers, Brian > - Pierre > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] Re: Help resolve massive performance regression in 2.7 vs 2.5 runtime
Half sized instances, so they may be half speed. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Pol Sent: Sunday, November 13, 2011 9:32 AM To: Google App Engine Subject: [google-appengine] Re: Help resolve massive performance regression in 2.7 vs 2.5 runtime Hi Brian, It helps, it's now at ~5s instead, but still at least 2x slower than on the 2.5 runtime. So on December 1st, the 50% discount for front-instances is gone. The idea is to compensate by switching to Python 2.7 with multithreading, but it looks like at this point it's a lose-lose situation: it runs more requests at the same time, but they take longer. We're mid- November already, do you guys think you'll have all of this working perfectly within 2 weeks? The 1.6 SDK which actually allows to run 2.7 locally was just released, so we're only starting to test now. Seems to me deferring the payment change 1 more month to January 1st 2012 wouldn't hurt :) On Nov 12, 10:09 pm, Brian Quinlan wrote: > Hi Pol, > > Thanks for getting back to me. > > On Sun, Nov 13, 2011 at 4:50 PM, Pol wrote: > > Hi Brian, > > > threadsafe is "true" > > There is a known issue where concurrent requests (enabled with > threadsafe) can be much slower than non-concurrent requests, > especially if the request is CPU-bound. > > You might want to set threadsafe to "false" and see if that fixes the problem. > > Cheers, > Brian > > > > > > > > > Here's an example password_hash for an original 8 characters password: > > > $2a$04$cbM2uHwDphIG3jFFRpq1mui5aVjevnDUwhvQ77S/WG/qvJMpiXAL6 > > > On Nov 12, 6:58 pm, Brian Quinlan wrote: > >> Hi Pol, > > >> On Sun, Nov 13, 2011 at 1:48 PM, Pol wrote: > >> > Hi, > > >> > Since switching to 2.7 runtime, logging in > >> > tohttp://www.everpix.com went from about a second to anywhere > >> > from 15s to 60s. I tracked it down to this single password checking line: > > >> > from bcrypt import bcrypt > >> > bcrypt.hashpw(password, self.password_hash) == self.password_hash > > >> What value are you using for "threadsafe" in your app.yaml? > > >> How large is self.password_hash? > > >> Cheers, > >> Brian > > >> > This comes from "a native Python implementation of the py-bcrypt > >> > package fromhttp://www.mindrot.org/projects/py-bcrypt/"; grabbed > >> > from here:https://github.com/erlichmen/py-bcrypt. > > >> > So what's happening here and how can we fix this? > > >> > Thanks, > > >> > - Pol > > >> > -- > >> > You received this message because you are subscribed to the Google Groups "Google App Engine" group. > >> > To post to this group, send email to google-appengine@googlegroups.com. > >> > To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. > >> > For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Help resolve massive performance regression in 2.7 vs 2.5 runtime
On Mon, Nov 14, 2011 at 12:19 PM, Pol wrote: > > But note that concurrent requests will *not* improve the utilization > > of CPU-bound requests. Running multiple threads on the same CPU just > > proportionally slows each thread down. > > That doesn't make sense: apps do a mix of CPU stuff and RPC stuff (and > possibly URL requests). What's the points of concurrent requests if it > slows down the CPU stuff while allowing to parallelize your RPC calls? > The end result will be the same number of instance as requests end up > taking longer. Isn't the scheduler supposed to watch all this and > ensure the CPU on each physical machine is not saturated? > > Only apps that do long poll URL requests and barely use the CPU would > benefit of concurrent requests then. > The vast majority of web applications (on any platform) spend the majority of their time blocked waiting for I/O. It's usually an order of magnitude difference - and this isn't long polling, it's just normal database access and whatnot. It sounds like your bcrypt library is intolerably slow in python, which creates a rare CPU bound situation. Either decrement the number of rounds or consider more efficient but still reasonably secure alternatives in this thread: http://stackoverflow.com/questions/7027196/how-can-i-use-bcrypt-scrypt-on-appengine-for-python Also, I +1 Nick's comment about DON'T roll your own. There are plenty of off-the-shelf options written by people who have spent significant portions of their educated life thinking about this problem. Jeff -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
