Re: [google-appengine] App Engine Scaling/Error Concern
Thanks for the suggestions. I had one more question - Although we're using sparse entity distribution, each of our entry is indexed on two primary properties, so we don't build any composite indexes. But the entities are retrieved through the basic index at a really fast rate, no writes, only reads, but they are really fast. Should I be worried about contention issues? Some entities might face 1000+ reads/sec retrieved though a query done on a property. But we retrieve only upto 30-50 entities per query. Thanks again! On Thu, Aug 12, 2010 at 1:32 AM, Ikai L (Google) ika...@google.com wrote: Measure your QPS, use AppStats, and be sure your requests that are user facing finish quickly. If the majority of your requests finish in under 800ms and return to the user, we'll spin up new instances for you. 400ms is best, however. That's 800ms of perceived time for the user - not total time (datastore calls are parallelized, so you could consume 2000ms of CPU time, but if your request returns to the user in 300ms you are okay). Do any work that will take longer in Task Queues or Cron when possible. On Wed, Aug 11, 2010 at 11:11 AM, Nikita Srivastava nikisri...@gmail.comwrote: Makes sense. Also, another user here pointed me to the 1.3.6 specs which says that custom error pages would be soon available. So the Quota Error be less of an issue, because we do get errors and we do show them, but the main issue was fear of sudden 'app engine' error when no user knows what app engine is. I hope 1.3.6 resolves this. As for the reason you mentioned, I think what we should do is set daily budget twice of what we paid for CPU previously, which was not cheaper I think ($0.08 per 1.2Ghz CPU hour). So I am planning to set this to $2000-2500 range as daily limit to avoid the Quota errors to legit users. Any other things that you can suggest that I should do? Thanks! On Wed, Aug 11, 2010 at 8:13 PM, Nick Johnson (Google) nick.john...@google.com wrote: Hi Nikita, On Wed, Aug 11, 2010 at 7:01 AM, Nikita nikisri...@gmail.com wrote: Hello, We're trying to deploy a large project on app engine, and by this time, we have an app engine version in internal testing. However, we have a few concerns: First, the information this gentleman expresses: http://aralbalkan.com/1504 Here is a quote - You build an awesome new app on Google App Engine. You tell your friends. They tell 1,000 of their friends on Twitter who tell 1,000 of their friends and then, suddenly, you have all these developers hitting Google App Engine for the first time to see your app. Paradoxically, by doing that, they trigger the intelligent throttling feature in Google App Engine which freaks out and shuts down your app with an Over Quota error -- effectively making the Over Quota message the first impression most of your audience has of Google App Engine. To what extent is this true? As paying customers with a sizable budget (We're paying over $35,000 a month on our current host, bandwidth exclusive), this is completely unacceptable. There's no intelligent throttling. Every app has quotas, determined by the amount of budget you allocate to each resource (or a fixed amount, in the case of free apps), and the quotas are accounted for both on a daily basis and in smaller bins. In other words, your traffic can vary, but we won't let your app consume its entire budget before we start serving over quota errors. I hope you can see why this is: we assume you don't want a single spike of traffic, for example because your app is being abused, to be able to consume your entire quota for the day, leaving your app serving 'out of quota' errors for the rest of the day! I assumed that this might be true only for free accounts, but then one of our engineers found this - http://groups.google.com/group/google-appengine/browse_thread/thread/32876e345d075878/e2ec2e4fdd7ca3e0 This is a paying customer begging for your mercy to scale. He claims he gets a Out of Quota error in 1 of 6 requests. This shows he isn't really out o his daily Quota, but there is something else that's going wrong. As you can see from the thread, the user was only experiencing issues because some of their requests were taking too long to complete. We impose limits on how many instances of an app we will schedule if the app takes too long to respond, because slow apps don't scale very well and would take disproportionate amounts of resources. -Nick Johnson It would be great i somebody from Google can comment on the validity o the above concerns. We don't want to shift our entire business to app engine and have our customers watch the Out of Quota - App engine errors. Frankly, this would be like BSOD of the web 2.0 world. :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this
Re: [google-appengine] App Engine Scaling/Error Concern
Hi Nikita, On Thu, Aug 12, 2010 at 3:02 PM, Nikita Srivastava nikisri...@gmail.comwrote: Thanks for the suggestions. I had one more question - Although we're using sparse entity distribution, each of our entry is indexed on two primary properties, so we don't build any composite indexes. But the entities are retrieved through the basic index at a really fast rate, no writes, only reads, but they are really fast. Should I be worried about contention issues? Some entities might face 1000+ reads/sec retrieved though a query done on a property. But we retrieve only upto 30-50 entities per query. No, this won't be a problem. Bigtable tabletservers cache frequently read data in memory, and split tablets across more servers if they're more popular - so it would take a truly astounding read rate on a single row to even begin to cause an issue, at which point you should be using Memcache anyway! -Nick Johnson Thanks again! On Thu, Aug 12, 2010 at 1:32 AM, Ikai L (Google) ika...@google.comwrote: Measure your QPS, use AppStats, and be sure your requests that are user facing finish quickly. If the majority of your requests finish in under 800ms and return to the user, we'll spin up new instances for you. 400ms is best, however. That's 800ms of perceived time for the user - not total time (datastore calls are parallelized, so you could consume 2000ms of CPU time, but if your request returns to the user in 300ms you are okay). Do any work that will take longer in Task Queues or Cron when possible. On Wed, Aug 11, 2010 at 11:11 AM, Nikita Srivastava nikisri...@gmail.com wrote: Makes sense. Also, another user here pointed me to the 1.3.6 specs which says that custom error pages would be soon available. So the Quota Error be less of an issue, because we do get errors and we do show them, but the main issue was fear of sudden 'app engine' error when no user knows what app engine is. I hope 1.3.6 resolves this. As for the reason you mentioned, I think what we should do is set daily budget twice of what we paid for CPU previously, which was not cheaper I think ($0.08 per 1.2Ghz CPU hour). So I am planning to set this to $2000-2500 range as daily limit to avoid the Quota errors to legit users. Any other things that you can suggest that I should do? Thanks! On Wed, Aug 11, 2010 at 8:13 PM, Nick Johnson (Google) nick.john...@google.com wrote: Hi Nikita, On Wed, Aug 11, 2010 at 7:01 AM, Nikita nikisri...@gmail.com wrote: Hello, We're trying to deploy a large project on app engine, and by this time, we have an app engine version in internal testing. However, we have a few concerns: First, the information this gentleman expresses: http://aralbalkan.com/1504 Here is a quote - You build an awesome new app on Google App Engine. You tell your friends. They tell 1,000 of their friends on Twitter who tell 1,000 of their friends and then, suddenly, you have all these developers hitting Google App Engine for the first time to see your app. Paradoxically, by doing that, they trigger the intelligent throttling feature in Google App Engine which freaks out and shuts down your app with an Over Quota error -- effectively making the Over Quota message the first impression most of your audience has of Google App Engine. To what extent is this true? As paying customers with a sizable budget (We're paying over $35,000 a month on our current host, bandwidth exclusive), this is completely unacceptable. There's no intelligent throttling. Every app has quotas, determined by the amount of budget you allocate to each resource (or a fixed amount, in the case of free apps), and the quotas are accounted for both on a daily basis and in smaller bins. In other words, your traffic can vary, but we won't let your app consume its entire budget before we start serving over quota errors. I hope you can see why this is: we assume you don't want a single spike of traffic, for example because your app is being abused, to be able to consume your entire quota for the day, leaving your app serving 'out of quota' errors for the rest of the day! I assumed that this might be true only for free accounts, but then one of our engineers found this - http://groups.google.com/group/google-appengine/browse_thread/thread/32876e345d075878/e2ec2e4fdd7ca3e0 This is a paying customer begging for your mercy to scale. He claims he gets a Out of Quota error in 1 of 6 requests. This shows he isn't really out o his daily Quota, but there is something else that's going wrong. As you can see from the thread, the user was only experiencing issues because some of their requests were taking too long to complete. We impose limits on how many instances of an app we will schedule if the app takes too long to respond, because slow apps don't scale very well and would take disproportionate amounts of resources. -Nick Johnson It would be great i somebody from Google can comment on the
[google-appengine] App Engine Scaling/Error Concern
Hello, We're trying to deploy a large project on app engine, and by this time, we have an app engine version in internal testing. However, we have a few concerns: First, the information this gentleman expresses: http://aralbalkan.com/1504 Here is a quote - You build an awesome new app on Google App Engine. You tell your friends. They tell 1,000 of their friends on Twitter who tell 1,000 of their friends and then, suddenly, you have all these developers hitting Google App Engine for the first time to see your app. Paradoxically, by doing that, they trigger the intelligent throttling feature in Google App Engine which freaks out and shuts down your app with an Over Quota error -- effectively making the Over Quota message the first impression most of your audience has of Google App Engine. To what extent is this true? As paying customers with a sizable budget (We're paying over $35,000 a month on our current host, bandwidth exclusive), this is completely unacceptable. I assumed that this might be true only for free accounts, but then one of our engineers found this - http://groups.google.com/group/google-appengine/browse_thread/thread/32876e345d075878/e2ec2e4fdd7ca3e0 This is a paying customer begging for your mercy to scale. He claims he gets a Out of Quota error in 1 of 6 requests. This shows he isn't really out o his daily Quota, but there is something else that's going wrong. It would be great i somebody from Google can comment on the validity o the above concerns. We don't want to shift our entire business to app engine and have our customers watch the Out of Quota - App engine errors. Frankly, this would be like BSOD of the web 2.0 world. :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine Scaling/Error Concern
Hi Nikita, On Wed, Aug 11, 2010 at 7:01 AM, Nikita nikisri...@gmail.com wrote: Hello, We're trying to deploy a large project on app engine, and by this time, we have an app engine version in internal testing. However, we have a few concerns: First, the information this gentleman expresses: http://aralbalkan.com/1504 Here is a quote - You build an awesome new app on Google App Engine. You tell your friends. They tell 1,000 of their friends on Twitter who tell 1,000 of their friends and then, suddenly, you have all these developers hitting Google App Engine for the first time to see your app. Paradoxically, by doing that, they trigger the intelligent throttling feature in Google App Engine which freaks out and shuts down your app with an Over Quota error -- effectively making the Over Quota message the first impression most of your audience has of Google App Engine. To what extent is this true? As paying customers with a sizable budget (We're paying over $35,000 a month on our current host, bandwidth exclusive), this is completely unacceptable. There's no intelligent throttling. Every app has quotas, determined by the amount of budget you allocate to each resource (or a fixed amount, in the case of free apps), and the quotas are accounted for both on a daily basis and in smaller bins. In other words, your traffic can vary, but we won't let your app consume its entire budget before we start serving over quota errors. I hope you can see why this is: we assume you don't want a single spike of traffic, for example because your app is being abused, to be able to consume your entire quota for the day, leaving your app serving 'out of quota' errors for the rest of the day! I assumed that this might be true only for free accounts, but then one of our engineers found this - http://groups.google.com/group/google-appengine/browse_thread/thread/32876e345d075878/e2ec2e4fdd7ca3e0 This is a paying customer begging for your mercy to scale. He claims he gets a Out of Quota error in 1 of 6 requests. This shows he isn't really out o his daily Quota, but there is something else that's going wrong. As you can see from the thread, the user was only experiencing issues because some of their requests were taking too long to complete. We impose limits on how many instances of an app we will schedule if the app takes too long to respond, because slow apps don't scale very well and would take disproportionate amounts of resources. -Nick Johnson It would be great i somebody from Google can comment on the validity o the above concerns. We don't want to shift our entire business to app engine and have our customers watch the Out of Quota - App engine errors. Frankly, this would be like BSOD of the web 2.0 world. :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] App Engine Scaling/Error Concern
Makes sense. Also, another user here pointed me to the 1.3.6 specs which says that custom error pages would be soon available. So the Quota Error be less of an issue, because we do get errors and we do show them, but the main issue was fear of sudden 'app engine' error when no user knows what app engine is. I hope 1.3.6 resolves this. As for the reason you mentioned, I think what we should do is set daily budget twice of what we paid for CPU previously, which was not cheaper I think ($0.08 per 1.2Ghz CPU hour). So I am planning to set this to $2000-2500 range as daily limit to avoid the Quota errors to legit users. Any other things that you can suggest that I should do? Thanks! On Wed, Aug 11, 2010 at 8:13 PM, Nick Johnson (Google) nick.john...@google.com wrote: Hi Nikita, On Wed, Aug 11, 2010 at 7:01 AM, Nikita nikisri...@gmail.com wrote: Hello, We're trying to deploy a large project on app engine, and by this time, we have an app engine version in internal testing. However, we have a few concerns: First, the information this gentleman expresses: http://aralbalkan.com/1504 Here is a quote - You build an awesome new app on Google App Engine. You tell your friends. They tell 1,000 of their friends on Twitter who tell 1,000 of their friends and then, suddenly, you have all these developers hitting Google App Engine for the first time to see your app. Paradoxically, by doing that, they trigger the intelligent throttling feature in Google App Engine which freaks out and shuts down your app with an Over Quota error -- effectively making the Over Quota message the first impression most of your audience has of Google App Engine. To what extent is this true? As paying customers with a sizable budget (We're paying over $35,000 a month on our current host, bandwidth exclusive), this is completely unacceptable. There's no intelligent throttling. Every app has quotas, determined by the amount of budget you allocate to each resource (or a fixed amount, in the case of free apps), and the quotas are accounted for both on a daily basis and in smaller bins. In other words, your traffic can vary, but we won't let your app consume its entire budget before we start serving over quota errors. I hope you can see why this is: we assume you don't want a single spike of traffic, for example because your app is being abused, to be able to consume your entire quota for the day, leaving your app serving 'out of quota' errors for the rest of the day! I assumed that this might be true only for free accounts, but then one of our engineers found this - http://groups.google.com/group/google-appengine/browse_thread/thread/32876e345d075878/e2ec2e4fdd7ca3e0 This is a paying customer begging for your mercy to scale. He claims he gets a Out of Quota error in 1 of 6 requests. This shows he isn't really out o his daily Quota, but there is something else that's going wrong. As you can see from the thread, the user was only experiencing issues because some of their requests were taking too long to complete. We impose limits on how many instances of an app we will schedule if the app takes too long to respond, because slow apps don't scale very well and would take disproportionate amounts of resources. -Nick Johnson It would be great i somebody from Google can comment on the validity o the above concerns. We don't want to shift our entire business to app engine and have our customers watch the Out of Quota - App engine errors. Frankly, this would be like BSOD of the web 2.0 world. :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at
Re: [google-appengine] App Engine Scaling/Error Concern
Measure your QPS, use AppStats, and be sure your requests that are user facing finish quickly. If the majority of your requests finish in under 800ms and return to the user, we'll spin up new instances for you. 400ms is best, however. That's 800ms of perceived time for the user - not total time (datastore calls are parallelized, so you could consume 2000ms of CPU time, but if your request returns to the user in 300ms you are okay). Do any work that will take longer in Task Queues or Cron when possible. On Wed, Aug 11, 2010 at 11:11 AM, Nikita Srivastava nikisri...@gmail.comwrote: Makes sense. Also, another user here pointed me to the 1.3.6 specs which says that custom error pages would be soon available. So the Quota Error be less of an issue, because we do get errors and we do show them, but the main issue was fear of sudden 'app engine' error when no user knows what app engine is. I hope 1.3.6 resolves this. As for the reason you mentioned, I think what we should do is set daily budget twice of what we paid for CPU previously, which was not cheaper I think ($0.08 per 1.2Ghz CPU hour). So I am planning to set this to $2000-2500 range as daily limit to avoid the Quota errors to legit users. Any other things that you can suggest that I should do? Thanks! On Wed, Aug 11, 2010 at 8:13 PM, Nick Johnson (Google) nick.john...@google.com wrote: Hi Nikita, On Wed, Aug 11, 2010 at 7:01 AM, Nikita nikisri...@gmail.com wrote: Hello, We're trying to deploy a large project on app engine, and by this time, we have an app engine version in internal testing. However, we have a few concerns: First, the information this gentleman expresses: http://aralbalkan.com/1504 Here is a quote - You build an awesome new app on Google App Engine. You tell your friends. They tell 1,000 of their friends on Twitter who tell 1,000 of their friends and then, suddenly, you have all these developers hitting Google App Engine for the first time to see your app. Paradoxically, by doing that, they trigger the intelligent throttling feature in Google App Engine which freaks out and shuts down your app with an Over Quota error -- effectively making the Over Quota message the first impression most of your audience has of Google App Engine. To what extent is this true? As paying customers with a sizable budget (We're paying over $35,000 a month on our current host, bandwidth exclusive), this is completely unacceptable. There's no intelligent throttling. Every app has quotas, determined by the amount of budget you allocate to each resource (or a fixed amount, in the case of free apps), and the quotas are accounted for both on a daily basis and in smaller bins. In other words, your traffic can vary, but we won't let your app consume its entire budget before we start serving over quota errors. I hope you can see why this is: we assume you don't want a single spike of traffic, for example because your app is being abused, to be able to consume your entire quota for the day, leaving your app serving 'out of quota' errors for the rest of the day! I assumed that this might be true only for free accounts, but then one of our engineers found this - http://groups.google.com/group/google-appengine/browse_thread/thread/32876e345d075878/e2ec2e4fdd7ca3e0 This is a paying customer begging for your mercy to scale. He claims he gets a Out of Quota error in 1 of 6 requests. This shows he isn't really out o his daily Quota, but there is something else that's going wrong. As you can see from the thread, the user was only experiencing issues because some of their requests were taking too long to complete. We impose limits on how many instances of an app we will schedule if the app takes too long to respond, because slow apps don't scale very well and would take disproportionate amounts of resources. -Nick Johnson It would be great i somebody from Google can comment on the validity o the above concerns. We don't want to shift our entire business to app engine and have our customers watch the Out of Quota - App engine errors. Frankly, this would be like BSOD of the web 2.0 world. :) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com . For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to
