mitnick, Can you explain more fully what the problem was, and how you were able to debug the problem once you knew to look at the logs vs the dashboard? After all the back and forth on the high cpu quota issue, I am curious about the particulars. Right now, this is how I understand it:
Warnings in the logs are based solely on the CPU used in Non API calls - so if I had a request that did nothing but try to factor a really large prime, I would probably get a lot of these warnings, and they would also trigger dashboard warnings? Warnings in the dashboard include CPU used in API calls, so if I had a request that did nothing but complicated datastore calls, it would get a dashboard warning, but *not* a log warning? If that's the case, I am guessing you (mitnick) had a request that was using a lot of runtime CPU but not necessarily a lot of API time? On Sep 26, 8:55 pm, mitnickcbc <[EMAIL PROTECTED]> wrote: > > Request denials for high CPU are based on the runtime CPU usage (which > > does not include the CPU used in API calls). Since the runtime CPU > > cost are the ones present in the logs, it is more useful to rely on > > the log information when trying to determine how best to reduce CPU > > load for specific requests. > > Thanks for the clarification! Now I'm able to find the bottleneck > and... it scales! > > On Sep 27, 2:00 am, Jeff S <[EMAIL PROTECTED]> wrote: > > > On Sep 26, 1:48 am, mitnickcbc <[EMAIL PROTECTED]> wrote: > > > > Some update about my latest findings. And these are my guesses, please > > > do correct me if I'm wrong. > > > > The quota denial for high amount CPU and the warning message "Many of > > > the requests to your application are taking a very long time. Please > > > optimize these requests." is calculated based on the number of warning > > > you get in your log saying that "This request used a high amount of > > > CPU, and was roughly 1.0 times over the average request CPU limit. > > > High CPU requests have a small quota, and if you exceed this quota, > > > your app will be temporarily disabled." > > > > Please note, the average CPU usage and related warning are very > > > misleading. The mcycles value displayed in log is also very different > > > from the x.x times value showed in warning message. I used to think > > > that request using beyond 1800 mcycles CPU will be considered as a > > > high CPU request since it shows red in Average CPU column. But the > > > latest finding is that it's not the case. One kind of my requests use > > > about 6K-7K mcycles in average CPU, but I seldom see a high CPU > > > warning in log for them. Again, this mcycles value seems to mean > > > nothing and the algorithm used to calculate the x.x times high CPU > > > warning really matters! > > > As you have noticed, there are differences in the CPU usage > > measurements presented in the logs and in the dashboard. This thread > > contains clarification which I've copied below: > > >http://groups.google.com/group/google-appengine/browse_frm/thread/f21... > > "Concerning CPU accounting overall - The warning threshold for the > > logs > > and for the dashboard actually differ. In the logs, warnings will > > only > > occur for high runtime CPU. The warnings on the dashboard are > > triggered for > > high overall CPU, including CPU used in the runtime and per API, most > > notably being the datastore API." > > > Request denials for high CPU are based on the runtime CPU usage (which > > does not include the CPU used in API calls). Since the runtime CPU > > cost are the ones present in the logs, it is more useful to rely on > > the log information when trying to determine how best to reduce CPU > > load for specific requests. > > > Happy coding, > > > Jeff > > > > With this in mind, the focus is back to the warning message itself to > > > figure out what's the algorithm. So there is an "average request CPU > > > limit" in this message. This value seems to be quite big and only my > > > requests that are doing some work which should be done by a background > > > scheduler exceeds it. The question is, is this a fixed value? Or a > > > dynamic value calculated based on all apps? Or a dynamic value > > > calculated based on the app itself? > > > > And the number of such warnings has a fixed quota that doesn't scale > > > with other quotas. If all my assumption is right here, the best option > > > to solve this issue in my mind is to make the high CPU quota scale as > > > other quotas. > > > > On Sep 26, 1:22 am, mitnickcbc <[EMAIL PROTECTED]> wrote: > > > > > Anyone has any idea how this warning message is calculated? "Many of > > > > the requests to your application are taking a very long time. Please > > > > optimize these requests." > > > > > It seems I meet quota denials every time this one shows up. I'm unable > > > > to tune anymore if I don't know how it is calculated. > > > > > There are several possible algorithms for this in my mind and I need > > > > to know which one is used to be able to move forward: > > > > 1. Based on the total number of high CPU requests in a certain period > > > > of time. And what's the CPU usage value to make a request counted in? > > > > 1200 mcycles? 1800, or anything else? > > > > 2. Based on total CPU consumed by these high CPU. > > > > 3. Based on response time. > > > > > However, no matter how much I tuned, it's just about time for me to > > > > reach the load limit. Since 30% of my write requests will never > > > > possible to fit the current high CPU bar. It also seems that the high > > > > CPU quota is not increased with the other quota. So say assume 10% of > > > > your requests are high CPU ones, and GAE only allows 1 high CPU > > > > request per second, then you will not be able to scale beyond 10 > > > > requests per second, no matter how much CPU quota you have. > > > > > There are 3 solutions in my mind: > > > > 1. Raise the high CPU bar to fit real world needs like raising it to > > > > 10K mcycles. > > > > 2. Make the high amount CPU quota scalable and configurable instead of > > > > a fixed number. Such as making it 30% of normal CPU quota. However, I > > > > would still suggest to not even fix the percentage since every > > > > application is different. Better to make it configurable by developer > > > > themselves. > > > > 3. Make high amount CPU quota same as other quotas like CPU and > > > > bandwidth. So developer can apply for it if they are running out of > > > > it. > > > > > I'm not sure which one is more doable to GAE, but there must be some > > > > solution to be happen otherwise it will not scale. > > > > > On Sep 21, 7:27 pm,mitnickcbc<[EMAIL PROTECTED]> wrote: > > > > > > I have filed a feature request for this issue, if you feel so please > > > > > star it. > > > > > >http://code.google.com/p/googleappengine/issues/detail?id=720 > > > > > > Here is the description: > > > > > > Well, I don't get the purpose for having a High Amount CPU Quota since > > > > > there is already a general CPU Quota. And there are quite a lot of > > > > > problems caused by this quota which restrict my application from > > > > > affording more load. And my application is far away from a 5 million > > > > > month PV app. > > > > > 1. Requests can become a high amount CPU one very easily. A little > > > > > complex request will consume more than 3K mcycles. And what do I mean > > > > > a little > > > > > complex? If you do more than 2 put, or you do a url fetch, or you do a > > > > > query based on an order to select more than 50 model, or you do a put > > > > > on a > > > > > model more than 20 fields, it is that complex. In my case, 80% of the > > > > > requests are that complex and I don't think it can be optimized > > > > > anymore > > > > > since I do need to do put and url fetch. > > > > > 2. Bad visibility to this quota. There is no way we can know when we > > > > > will run out of quota for this. All we got is the warning in admin > > > > > console that > > > > > says our requests are using high amount CPU quota and will soon run > > > > > out of it. > > > > > 3. Doesn't back to normal quickly. I'm using a script to continue > > > > > fetching my data from production and do some offline process. > > > > > Unfortunately I fetch > > > > > too much in one call that I try to get 100 models at a time and which > > > > > makes the request a high CPU one. Soon, I meet quota deny. Then I stop > > > > > my script, > > > > > but after 1.5 hours, I still see quota deny for requests from my > > > > > users. The "Many of the requests to your application are taking a very > > > > > long time. > > > > > Please optimize these requests." warning keeps there even I have > > > > > stopped the script for so long time. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
