> Request denials for high CPU are based on the runtime CPU usage (which > does not include the CPU used in API calls). Since the runtime CPU > cost are the ones present in the logs, it is more useful to rely on > the log information when trying to determine how best to reduce CPU > load for specific requests.
Thanks for the clarification! Now I'm able to find the bottleneck and... it scales! On Sep 27, 2:00 am, Jeff S <[EMAIL PROTECTED]> wrote: > On Sep 26, 1:48 am, mitnickcbc <[EMAIL PROTECTED]> wrote: > > > > > Some update about my latest findings. And these are my guesses, please > > do correct me if I'm wrong. > > > The quota denial for high amount CPU and the warning message "Many of > > the requests to your application are taking a very long time. Please > > optimize these requests." is calculated based on the number of warning > > you get in your log saying that "This request used a high amount of > > CPU, and was roughly 1.0 times over the average request CPU limit. > > High CPU requests have a small quota, and if you exceed this quota, > > your app will be temporarily disabled." > > > Please note, the average CPU usage and related warning are very > > misleading. The mcycles value displayed in log is also very different > > from the x.x times value showed in warning message. I used to think > > that request using beyond 1800 mcycles CPU will be considered as a > > high CPU request since it shows red in Average CPU column. But the > > latest finding is that it's not the case. One kind of my requests use > > about 6K-7K mcycles in average CPU, but I seldom see a high CPU > > warning in log for them. Again, this mcycles value seems to mean > > nothing and the algorithm used to calculate the x.x times high CPU > > warning really matters! > > As you have noticed, there are differences in the CPU usage > measurements presented in the logs and in the dashboard. This thread > contains clarification which I've copied below: > > http://groups.google.com/group/google-appengine/browse_frm/thread/f21... > "Concerning CPU accounting overall - The warning threshold for the > logs > and for the dashboard actually differ. In the logs, warnings will > only > occur for high runtime CPU. The warnings on the dashboard are > triggered for > high overall CPU, including CPU used in the runtime and per API, most > notably being the datastore API." > > Request denials for high CPU are based on the runtime CPU usage (which > does not include the CPU used in API calls). Since the runtime CPU > cost are the ones present in the logs, it is more useful to rely on > the log information when trying to determine how best to reduce CPU > load for specific requests. > > Happy coding, > > Jeff > > > > > With this in mind, the focus is back to the warning message itself to > > figure out what's the algorithm. So there is an "average request CPU > > limit" in this message. This value seems to be quite big and only my > > requests that are doing some work which should be done by a background > > scheduler exceeds it. The question is, is this a fixed value? Or a > > dynamic value calculated based on all apps? Or a dynamic value > > calculated based on the app itself? > > > And the number of such warnings has a fixed quota that doesn't scale > > with other quotas. If all my assumption is right here, the best option > > to solve this issue in my mind is to make the high CPU quota scale as > > other quotas. > > > On Sep 26, 1:22 am, mitnickcbc <[EMAIL PROTECTED]> wrote: > > > > Anyone has any idea how this warning message is calculated? "Many of > > > the requests to your application are taking a very long time. Please > > > optimize these requests." > > > > It seems I meet quota denials every time this one shows up. I'm unable > > > to tune anymore if I don't know how it is calculated. > > > > There are several possible algorithms for this in my mind and I need > > > to know which one is used to be able to move forward: > > > 1. Based on the total number of high CPU requests in a certain period > > > of time. And what's the CPU usage value to make a request counted in? > > > 1200 mcycles? 1800, or anything else? > > > 2. Based on total CPU consumed by these high CPU. > > > 3. Based on response time. > > > > However, no matter how much I tuned, it's just about time for me to > > > reach the load limit. Since 30% of my write requests will never > > > possible to fit the current high CPU bar. It also seems that the high > > > CPU quota is not increased with the other quota. So say assume 10% of > > > your requests are high CPU ones, and GAE only allows 1 high CPU > > > request per second, then you will not be able to scale beyond 10 > > > requests per second, no matter how much CPU quota you have. > > > > There are 3 solutions in my mind: > > > 1. Raise the high CPU bar to fit real world needs like raising it to > > > 10K mcycles. > > > 2. Make the high amount CPU quota scalable and configurable instead of > > > a fixed number. Such as making it 30% of normal CPU quota. However, I > > > would still suggest to not even fix the percentage since every > > > application is different. Better to make it configurable by developer > > > themselves. > > > 3. Make high amount CPU quota same as other quotas like CPU and > > > bandwidth. So developer can apply for it if they are running out of > > > it. > > > > I'm not sure which one is more doable to GAE, but there must be some > > > solution to be happen otherwise it will not scale. > > > > On Sep 21, 7:27 pm,mitnickcbc<[EMAIL PROTECTED]> wrote: > > > > > I have filed a feature request for this issue, if you feel so please > > > > star it. > > > > >http://code.google.com/p/googleappengine/issues/detail?id=720 > > > > > Here is the description: > > > > > Well, I don't get the purpose for having a High Amount CPU Quota since > > > > there is already a general CPU Quota. And there are quite a lot of > > > > problems caused by this quota which restrict my application from > > > > affording more load. And my application is far away from a 5 million > > > > month PV app. > > > > 1. Requests can become a high amount CPU one very easily. A little > > > > complex request will consume more than 3K mcycles. And what do I mean > > > > a little > > > > complex? If you do more than 2 put, or you do a url fetch, or you do a > > > > query based on an order to select more than 50 model, or you do a put > > > > on a > > > > model more than 20 fields, it is that complex. In my case, 80% of the > > > > requests are that complex and I don't think it can be optimized > > > > anymore > > > > since I do need to do put and url fetch. > > > > 2. Bad visibility to this quota. There is no way we can know when we > > > > will run out of quota for this. All we got is the warning in admin > > > > console that > > > > says our requests are using high amount CPU quota and will soon run > > > > out of it. > > > > 3. Doesn't back to normal quickly. I'm using a script to continue > > > > fetching my data from production and do some offline process. > > > > Unfortunately I fetch > > > > too much in one call that I try to get 100 models at a time and which > > > > makes the request a high CPU one. Soon, I meet quota deny. Then I stop > > > > my script, > > > > but after 1.5 hours, I still see quota deny for requests from my > > > > users. The "Many of the requests to your application are taking a very > > > > long time. > > > > Please optimize these requests." warning keeps there even I have > > > > stopped the script for so long time. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
