[google-appengine] How to gracefully handle datastore timeouts
Earlier this morning I had a situation where datastore reads were timing out. That's okay, and expected, given that I use a M/S datastore. However, the timeouts were of the order of 50 seconds, causing nearly 30 front-end instances to be spawned. My usual number of active front-end instances at that time of the day is about 5, peaking at 15 occasionally. This condition lasted only 3 minutes or so, and so, the cost impact was minimal. However, I can imagine that if this lasted an hour or more, I would incur a lot of costs while the downtime persists. I'm okay with such downtimes, as long as it only leads to my customers not being able to access my site. But if it also leads to unnecessary increases in costs, then it calls for further optimization. So, my loaded question is - how can I handle this with python2.5? Is python 2.7 the only answer? I imagine python2.7 will help because while a front-end is waiting for data store ops to complete, it can process other requests. But are there other ways of setting specific timeouts to datastore operations? So, if these operations are taking too long, I'd rather just return an error to the user, instead of letting my front-end run indefinitely. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] How to gracefully handle datastore timeouts
You can't use 2.7 unless you migrate to HR. Once you migrate to HR, the datastore read timeouts go away. And then you don't need to migrate to 2.7. -Joshua On Dec 2, 2011, at 10:37 AM, Rishi Arora wrote: > Earlier this morning I had a situation where datastore reads were timing out. > That's okay, and expected, given that I use a M/S datastore. However, the > timeouts were of the order of 50 seconds, causing nearly 30 front-end > instances to be spawned. My usual number of active front-end instances at > that time of the day is about 5, peaking at 15 occasionally. This condition > lasted only 3 minutes or so, and so, the cost impact was minimal. However, I > can imagine that if this lasted an hour or more, I would incur a lot of costs > while the downtime persists. I'm okay with such downtimes, as long as it > only leads to my customers not being able to access my site. But if it also > leads to unnecessary increases in costs, then it calls for further > optimization. So, my loaded question is - how can I handle this with > python2.5? Is python 2.7 the only answer? I imagine python2.7 will help > because while a front-end is waiting for data store ops to complete, it can > process other requests. But are there other ways of setting specific > timeouts to datastore operations? So, if these operations are taking too > long, I'd rather just return an error to the user, instead of letting my > front-end run indefinitely. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] How to gracefully handle datastore timeouts
:) So I'm SOL, without HR? On Fri, Dec 2, 2011 at 9:50 AM, Joshua Smith wrote: > You can't use 2.7 unless you migrate to HR. > > Once you migrate to HR, the datastore read timeouts go away. > > And then you don't need to migrate to 2.7. > > -Joshua > > On Dec 2, 2011, at 10:37 AM, Rishi Arora wrote: > > > Earlier this morning I had a situation where datastore reads were timing > out. That's okay, and expected, given that I use a M/S datastore. > However, the timeouts were of the order of 50 seconds, causing nearly 30 > front-end instances to be spawned. My usual number of active front-end > instances at that time of the day is about 5, peaking at 15 occasionally. > This condition lasted only 3 minutes or so, and so, the cost impact was > minimal. However, I can imagine that if this lasted an hour or more, I > would incur a lot of costs while the downtime persists. I'm okay with such > downtimes, as long as it only leads to my customers not being able to > access my site. But if it also leads to unnecessary increases in costs, > then it calls for further optimization. So, my loaded question is - how > can I handle this with python2.5? Is python 2.7 the only answer? I > imagine python2.7 will help because while a front-end is waiting for data > store ops to complete, it can process other requests. But are there other > ways of setting specific timeouts to datastore operations? So, if these > operations are taking too long, I'd rather just return an error to the > user, instead of letting my front-end run indefinitely. > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] How to gracefully handle datastore timeouts
I believe so. Google has made it pretty clear that M/S is deprecated and life on M/S will continue to get worse and worse. On Dec 2, 2011, at 10:54 AM, Rishi Arora wrote: > :) So I'm SOL, without HR? > > On Fri, Dec 2, 2011 at 9:50 AM, Joshua Smith wrote: > You can't use 2.7 unless you migrate to HR. > > Once you migrate to HR, the datastore read timeouts go away. > > And then you don't need to migrate to 2.7. > > -Joshua > > On Dec 2, 2011, at 10:37 AM, Rishi Arora wrote: > > > Earlier this morning I had a situation where datastore reads were timing > > out. That's okay, and expected, given that I use a M/S datastore. > > However, the timeouts were of the order of 50 seconds, causing nearly 30 > > front-end instances to be spawned. My usual number of active front-end > > instances at that time of the day is about 5, peaking at 15 occasionally. > > This condition lasted only 3 minutes or so, and so, the cost impact was > > minimal. However, I can imagine that if this lasted an hour or more, I > > would incur a lot of costs while the downtime persists. I'm okay with such > > downtimes, as long as it only leads to my customers not being able to > > access my site. But if it also leads to unnecessary increases in costs, > > then it calls for further optimization. So, my loaded question is - how > > can I handle this with python2.5? Is python 2.7 the only answer? I > > imagine python2.7 will help because while a front-end is waiting for data > > store ops to complete, it can process other requests. But are there other > > ways of setting specific timeouts to datastore operations? So, if these > > operations are taking too long, I'd rather just return an error to the > > user, instead of letting my front-end run indefinitely. > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appengine@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] How to gracefully handle datastore timeouts
Based on this: http://code.google.com/appengine/docs/python/datastore/hr/ I don't see any strong language indicating imminent deprecation. Although the fact that python2.7 does not support M/S might hint at that. If Google does indeed want to deprecate M/S, why wouldn't they just say it, and give some fixed timeline - 1 month, 6 months, 1 year, whatever. But why hint it, why not just say it? Since I don't see a strong reason for Google to drop subtle hints, I assumed there's no plan to deprecate M/S, and I also assumed that eventually when python2.5 is phased out, python2.7 will support M/S. I don't see any technical reason why python2.7 can't support M/S either. I can achieve nearly the same level of concurrency with Python2.5 (albeit at a higher cost) that I can with Python2.7, with regards to datastore operations. Can someone at Google provide some direction on long-term support (at least a year or so) for M/S? On Fri, Dec 2, 2011 at 10:07 AM, Joshua Smith wrote: > I believe so. > > Google has made it pretty clear that M/S is deprecated and life on M/S > will continue to get worse and worse. > > On Dec 2, 2011, at 10:54 AM, Rishi Arora wrote: > > :) So I'm SOL, without HR? > > On Fri, Dec 2, 2011 at 9:50 AM, Joshua Smith wrote: > >> You can't use 2.7 unless you migrate to HR. >> >> Once you migrate to HR, the datastore read timeouts go away. >> >> And then you don't need to migrate to 2.7. >> >> -Joshua >> >> On Dec 2, 2011, at 10:37 AM, Rishi Arora wrote: >> >> > Earlier this morning I had a situation where datastore reads were >> timing out. That's okay, and expected, given that I use a M/S datastore. >> However, the timeouts were of the order of 50 seconds, causing nearly 30 >> front-end instances to be spawned. My usual number of active front-end >> instances at that time of the day is about 5, peaking at 15 occasionally. >> This condition lasted only 3 minutes or so, and so, the cost impact was >> minimal. However, I can imagine that if this lasted an hour or more, I >> would incur a lot of costs while the downtime persists. I'm okay with such >> downtimes, as long as it only leads to my customers not being able to >> access my site. But if it also leads to unnecessary increases in costs, >> then it calls for further optimization. So, my loaded question is - how >> can I handle this with python2.5? Is python 2.7 the only answer? I >> imagine python2.7 will help because while a front-end is waiting for data >> store ops to complete, it can process other requests. But are there other >> ways of setting specific timeouts to datastore operations? So, if these >> operations are taking too long, I'd rather just return an error to the >> user, instead of letting my front-end run indefinitely. >> > >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Google App Engine" group. >> > To post to this group, send email to google-appengine@googlegroups.com. >> > To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> > For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] How to gracefully handle datastore timeouts
As far as I know, there are no plans to shut down M/S within the next 12 months. This is NOT an official announcement, so if this does change later (I don't think it will), just know that it's a gut feeling. When there is a plan to official deprecate master/slave, we will announce it. Many new features will be HRD only, however. Python 2.7 is one of them. Full text search *may* be one of them. There are no plans to support Python 2.7 for master/slave applications, and I don't see this changing. I strongly recommend migrating to HRD or at least starting to investigate the possibility. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Dec 2, 2011 at 9:03 AM, Rishi Arora wrote: > Based on this: > http://code.google.com/appengine/docs/python/datastore/hr/ > > I don't see any strong language indicating imminent deprecation. Although > the fact that python2.7 does not support M/S might hint at that. If Google > does indeed want to deprecate M/S, why wouldn't they just say it, and give > some fixed timeline - 1 month, 6 months, 1 year, whatever. But why hint > it, why not just say it? Since I don't see a strong reason for Google to > drop subtle hints, I assumed there's no plan to deprecate M/S, and I also > assumed that eventually when python2.5 is phased out, python2.7 will > support M/S. I don't see any technical reason why python2.7 can't support > M/S either. I can achieve nearly the same level of concurrency with > Python2.5 (albeit at a higher cost) that I can with Python2.7, with regards > to datastore operations. > > Can someone at Google provide some direction on long-term support (at > least a year or so) for M/S? > > > On Fri, Dec 2, 2011 at 10:07 AM, Joshua Smith wrote: > >> I believe so. >> >> Google has made it pretty clear that M/S is deprecated and life on M/S >> will continue to get worse and worse. >> >> On Dec 2, 2011, at 10:54 AM, Rishi Arora wrote: >> >> :) So I'm SOL, without HR? >> >> On Fri, Dec 2, 2011 at 9:50 AM, Joshua Smith wrote: >> >>> You can't use 2.7 unless you migrate to HR. >>> >>> Once you migrate to HR, the datastore read timeouts go away. >>> >>> And then you don't need to migrate to 2.7. >>> >>> -Joshua >>> >>> On Dec 2, 2011, at 10:37 AM, Rishi Arora wrote: >>> >>> > Earlier this morning I had a situation where datastore reads were >>> timing out. That's okay, and expected, given that I use a M/S datastore. >>> However, the timeouts were of the order of 50 seconds, causing nearly 30 >>> front-end instances to be spawned. My usual number of active front-end >>> instances at that time of the day is about 5, peaking at 15 occasionally. >>> This condition lasted only 3 minutes or so, and so, the cost impact was >>> minimal. However, I can imagine that if this lasted an hour or more, I >>> would incur a lot of costs while the downtime persists. I'm okay with such >>> downtimes, as long as it only leads to my customers not being able to >>> access my site. But if it also leads to unnecessary increases in costs, >>> then it calls for further optimization. So, my loaded question is - how >>> can I handle this with python2.5? Is python 2.7 the only answer? I >>> imagine python2.7 will help because while a front-end is waiting for data >>> store ops to complete, it can process other requests. But are there other >>> ways of setting specific timeouts to datastore operations? So, if these >>> operations are taking too long, I'd rather just return an error to the >>> user, instead of letting my front-end run indefinitely. >>> > >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> > To post to this group, send email to google-appengine@googlegroups.com >>> . >>> > To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> > For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To post to this group, send email to google-appengine@googlegroups.com. >>> To unsubscribe from this group, send email to >>> google-appengine+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >
Re: [google-appengine] How to gracefully handle datastore timeouts
To answer the original question: - As a general best practice, you almost always want to set a timeout for any call that might be frequent: http://code.google.com/appengine/docs/python/datastore/queries.html#Setting_the_Read_Policy_and_Datastore_Call_Deadline This is true for App Engine as well as any application outside of App Engine. You degrade gracefully by telling the user to retry or showing only a subset of data. It sucks, but at this point you have to decide between making the user wait a long time or showing partial results and doing an exponential backoff. - Use exponential backoffs if you tell your client to retry. You might show something to the user that says "loading, please wait!", but you're going to be in hot water if you just say "retry after 1 second, retry after 1 second, retry after 1 second" because these will eventually pile up. You want to do something like "retry after 2 seconds + some fuzz factor, retry after 4 seconds plus some fuzz factor, and so forth". - I've been part of a project where there were many administrative "knobs" that we could flip to gracefully degrade and turn off expensive features. This is particularly useful when you push a new version because you can narrow in on the component that seems to be causing your stack to break. It's also nice because you can turn something off, gather with your team and figure out what the plan of attack is without a full rollback. The tradeoff is that you have to build it, and it's non-trivial to have this all over your code, so you need to figure out which parts tend to be more complex than others. In an ideal world, you always know what component does what in your app, but in reality, you usually don't know that there is a weird code path that might invoke the most expensive method call over and over. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Dec 2, 2011 at 11:31 AM, Ikai Lan (Google) wrote: > As far as I know, there are no plans to shut down M/S within the next 12 > months. This is NOT an official announcement, so if this does change later > (I don't think it will), just know that it's a gut feeling. When there is a > plan to official deprecate master/slave, we will announce it. > > Many new features will be HRD only, however. Python 2.7 is one of > them. Full text search *may* be one of them. There are no plans to support > Python 2.7 for master/slave applications, and I don't see this changing. > > I strongly recommend migrating to HRD or at least starting to investigate > the possibility. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > > On Fri, Dec 2, 2011 at 9:03 AM, Rishi Arora wrote: > >> Based on this: >> http://code.google.com/appengine/docs/python/datastore/hr/ >> >> I don't see any strong language indicating imminent deprecation. >> Although the fact that python2.7 does not support M/S might hint at that. >> If Google does indeed want to deprecate M/S, why wouldn't they just say >> it, and give some fixed timeline - 1 month, 6 months, 1 year, whatever. >> But why hint it, why not just say it? Since I don't see a strong reason >> for Google to drop subtle hints, I assumed there's no plan to deprecate >> M/S, and I also assumed that eventually when python2.5 is phased out, >> python2.7 will support M/S. I don't see any technical reason why python2.7 >> can't support M/S either. I can achieve nearly the same level of >> concurrency with Python2.5 (albeit at a higher cost) that I can with >> Python2.7, with regards to datastore operations. >> >> Can someone at Google provide some direction on long-term support (at >> least a year or so) for M/S? >> >> >> On Fri, Dec 2, 2011 at 10:07 AM, Joshua Smith >> wrote: >> >>> I believe so. >>> >>> Google has made it pretty clear that M/S is deprecated and life on M/S >>> will continue to get worse and worse. >>> >>> On Dec 2, 2011, at 10:54 AM, Rishi Arora wrote: >>> >>> :) So I'm SOL, without HR? >>> >>> On Fri, Dec 2, 2011 at 9:50 AM, Joshua Smith >>> wrote: >>> You can't use 2.7 unless you migrate to HR. Once you migrate to HR, the datastore read timeouts go away. And then you don't need to migrate to 2.7. -Joshua On Dec 2, 2011, at 10:37 AM, Rishi Arora wrote: > Earlier this morning I had a situation where datastore reads were timing out. That's okay, and expected, given that I use a M/S datastore. However, the timeouts were of the order of 50 seconds, causing nearly 30 front-end instances to be spawned. My usual number of active front-end instances at that time of the day is about 5, peaking at 15 occasionally. This condition lasted only 3 minutes or so, and so, the cost impact was minimal. However, I can imagine that if this lasted an hour or more, I would incur a lot of costs while the downtime persists. I'm okay with such downtime
Re: [google-appengine] How to gracefully handle datastore timeouts
I just thought of something kind of cool: a real time knob that allows you to change the deadlines! -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Dec 2, 2011 at 11:37 AM, Ikai Lan (Google) wrote: > To answer the original question: > > - As a general best practice, you almost always want to set a timeout for > any call that might be frequent: > > > http://code.google.com/appengine/docs/python/datastore/queries.html#Setting_the_Read_Policy_and_Datastore_Call_Deadline > > This is true for App Engine as well as any application outside of App > Engine. You degrade gracefully by telling the user to retry or showing only > a subset of data. It sucks, but at this point you have to decide between > making the user wait a long time or showing partial results and doing an > exponential backoff. > > - Use exponential backoffs if you tell your client to retry. You might > show something to the user that says "loading, please wait!", but you're > going to be in hot water if you just say "retry after 1 second, retry after > 1 second, retry after 1 second" because these will eventually pile up. You > want to do something like "retry after 2 seconds + some fuzz factor, retry > after 4 seconds plus some fuzz factor, and so forth". > > - I've been part of a project where there were many administrative "knobs" > that we could flip to gracefully degrade and turn off expensive features. > This is particularly useful when you push a new version because you can > narrow in on the component that seems to be causing your stack to break. > It's also nice because you can turn something off, gather with your team > and figure out what the plan of attack is without a full rollback. The > tradeoff is that you have to build it, and it's non-trivial to have this > all over your code, so you need to figure out which parts tend to be more > complex than others. In an ideal world, you always know what component does > what in your app, but in reality, you usually don't know that there is a > weird code path that might invoke the most expensive method call over and > over. > > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > > On Fri, Dec 2, 2011 at 11:31 AM, Ikai Lan (Google) wrote: > >> As far as I know, there are no plans to shut down M/S within the next 12 >> months. This is NOT an official announcement, so if this does change later >> (I don't think it will), just know that it's a gut feeling. When there is a >> plan to official deprecate master/slave, we will announce it. >> >> Many new features will be HRD only, however. Python 2.7 is one of >> them. Full text search *may* be one of them. There are no plans to support >> Python 2.7 for master/slave applications, and I don't see this changing. >> >> I strongly recommend migrating to HRD or at least starting to investigate >> the possibility. >> >> -- >> Ikai Lan >> Developer Programs Engineer, Google App Engine >> plus.ikailan.com | twitter.com/ikai >> >> >> >> On Fri, Dec 2, 2011 at 9:03 AM, Rishi Arora wrote: >> >>> Based on this: >>> http://code.google.com/appengine/docs/python/datastore/hr/ >>> >>> I don't see any strong language indicating imminent deprecation. >>> Although the fact that python2.7 does not support M/S might hint at that. >>> If Google does indeed want to deprecate M/S, why wouldn't they just say >>> it, and give some fixed timeline - 1 month, 6 months, 1 year, whatever. >>> But why hint it, why not just say it? Since I don't see a strong reason >>> for Google to drop subtle hints, I assumed there's no plan to deprecate >>> M/S, and I also assumed that eventually when python2.5 is phased out, >>> python2.7 will support M/S. I don't see any technical reason why python2.7 >>> can't support M/S either. I can achieve nearly the same level of >>> concurrency with Python2.5 (albeit at a higher cost) that I can with >>> Python2.7, with regards to datastore operations. >>> >>> Can someone at Google provide some direction on long-term support (at >>> least a year or so) for M/S? >>> >>> >>> On Fri, Dec 2, 2011 at 10:07 AM, Joshua Smith >>> wrote: >>> I believe so. Google has made it pretty clear that M/S is deprecated and life on M/S will continue to get worse and worse. On Dec 2, 2011, at 10:54 AM, Rishi Arora wrote: :) So I'm SOL, without HR? On Fri, Dec 2, 2011 at 9:50 AM, Joshua Smith wrote: > You can't use 2.7 unless you migrate to HR. > > Once you migrate to HR, the datastore read timeouts go away. > > And then you don't need to migrate to 2.7. > > -Joshua > > On Dec 2, 2011, at 10:37 AM, Rishi Arora wrote: > > > Earlier this morning I had a situation where datastore reads were > timing out. That's okay, and expected, given that I use a M/S datastore. > However, the timeouts were of the order of 50 seconds, causing nearly 30 >>
Re: [google-appengine] How to gracefully handle datastore timeouts
Ikai, thank you for a comprehensive reply. I had no idea you could set deadlines for datastore reads, even for Master/Slave db reads. This would be my temporary solution until I eventually migrate to HRD. I have already started looking into it, with the toughest part being appropriately managing user experience during the downtime and during the changeover to the new app-id. My users use an Android app and Google Accounts API in conjunction with Android AccountManager, to interface with my GAE system, and they could be running multiple versions of my app at any given time. Thanks again - Rishi On Fri, Dec 2, 2011 at 1:38 PM, Ikai Lan (Google) wrote: > I just thought of something kind of cool: a real time knob that allows you > to change the deadlines! > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > plus.ikailan.com | twitter.com/ikai > > > > On Fri, Dec 2, 2011 at 11:37 AM, Ikai Lan (Google) wrote: > >> To answer the original question: >> >> - As a general best practice, you almost always want to set a timeout for >> any call that might be frequent: >> >> >> http://code.google.com/appengine/docs/python/datastore/queries.html#Setting_the_Read_Policy_and_Datastore_Call_Deadline >> >> This is true for App Engine as well as any application outside of App >> Engine. You degrade gracefully by telling the user to retry or showing only >> a subset of data. It sucks, but at this point you have to decide between >> making the user wait a long time or showing partial results and doing an >> exponential backoff. >> >> - Use exponential backoffs if you tell your client to retry. You might >> show something to the user that says "loading, please wait!", but you're >> going to be in hot water if you just say "retry after 1 second, retry after >> 1 second, retry after 1 second" because these will eventually pile up. You >> want to do something like "retry after 2 seconds + some fuzz factor, retry >> after 4 seconds plus some fuzz factor, and so forth". >> >> - I've been part of a project where there were many administrative >> "knobs" that we could flip to gracefully degrade and turn off expensive >> features. This is particularly useful when you push a new version because >> you can narrow in on the component that seems to be causing your stack to >> break. It's also nice because you can turn something off, gather with your >> team and figure out what the plan of attack is without a full rollback. The >> tradeoff is that you have to build it, and it's non-trivial to have this >> all over your code, so you need to figure out which parts tend to be more >> complex than others. In an ideal world, you always know what component does >> what in your app, but in reality, you usually don't know that there is a >> weird code path that might invoke the most expensive method call over and >> over. >> >> >> -- >> Ikai Lan >> Developer Programs Engineer, Google App Engine >> plus.ikailan.com | twitter.com/ikai >> >> >> >> On Fri, Dec 2, 2011 at 11:31 AM, Ikai Lan (Google) wrote: >> >>> As far as I know, there are no plans to shut down M/S within the next 12 >>> months. This is NOT an official announcement, so if this does change later >>> (I don't think it will), just know that it's a gut feeling. When there is a >>> plan to official deprecate master/slave, we will announce it. >>> >>> Many new features will be HRD only, however. Python 2.7 is one of >>> them. Full text search *may* be one of them. There are no plans to support >>> Python 2.7 for master/slave applications, and I don't see this changing. >>> >>> I strongly recommend migrating to HRD or at least starting to >>> investigate the possibility. >>> >>> -- >>> Ikai Lan >>> Developer Programs Engineer, Google App Engine >>> plus.ikailan.com | twitter.com/ikai >>> >>> >>> >>> On Fri, Dec 2, 2011 at 9:03 AM, Rishi Arora >>> wrote: >>> Based on this: http://code.google.com/appengine/docs/python/datastore/hr/ I don't see any strong language indicating imminent deprecation. Although the fact that python2.7 does not support M/S might hint at that. If Google does indeed want to deprecate M/S, why wouldn't they just say it, and give some fixed timeline - 1 month, 6 months, 1 year, whatever. But why hint it, why not just say it? Since I don't see a strong reason for Google to drop subtle hints, I assumed there's no plan to deprecate M/S, and I also assumed that eventually when python2.5 is phased out, python2.7 will support M/S. I don't see any technical reason why python2.7 can't support M/S either. I can achieve nearly the same level of concurrency with Python2.5 (albeit at a higher cost) that I can with Python2.7, with regards to datastore operations. Can someone at Google provide some direction on long-term support (at least a year or so) for M/S? On Fri, Dec 2, 2011 at 10:07 AM, Joshua Smith >>> > wrote: > I believe so. > > Goo
Re: [google-appengine] How to gracefully handle datastore timeouts
You can also set the consistency guarantees to EVENTUALLY_CONSISTENT. Even on master/slave datastore, this fails over to a read-slave. The tradeoff is you might have consistency issues. Here's a pretty good read about consistency vs. availability tradeoffs: http://codahale.com/you-cant-sacrifice-partition-tolerance/ It's good to know this stuff so you have it in your back pocket. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Dec 2, 2011 at 11:49 AM, Rishi Arora wrote: > Ikai, thank you for a comprehensive reply. I had no idea you could set > deadlines for datastore reads, even for Master/Slave db reads. This would > be my temporary solution until I eventually migrate to HRD. I have already > started looking into it, with the toughest part being appropriately > managing user experience during the downtime and during the changeover to > the new app-id. My users use an Android app and Google Accounts API in > conjunction with Android AccountManager, to interface with my GAE system, > and they could be running multiple versions of my app at any given time. > > Thanks again > - Rishi > > > On Fri, Dec 2, 2011 at 1:38 PM, Ikai Lan (Google) wrote: > >> I just thought of something kind of cool: a real time knob that allows >> you to change the deadlines! >> >> -- >> Ikai Lan >> Developer Programs Engineer, Google App Engine >> plus.ikailan.com | twitter.com/ikai >> >> >> >> On Fri, Dec 2, 2011 at 11:37 AM, Ikai Lan (Google) wrote: >> >>> To answer the original question: >>> >>> - As a general best practice, you almost always want to set a timeout >>> for any call that might be frequent: >>> >>> >>> http://code.google.com/appengine/docs/python/datastore/queries.html#Setting_the_Read_Policy_and_Datastore_Call_Deadline >>> >>> This is true for App Engine as well as any application outside of App >>> Engine. You degrade gracefully by telling the user to retry or showing only >>> a subset of data. It sucks, but at this point you have to decide between >>> making the user wait a long time or showing partial results and doing an >>> exponential backoff. >>> >>> - Use exponential backoffs if you tell your client to retry. You might >>> show something to the user that says "loading, please wait!", but you're >>> going to be in hot water if you just say "retry after 1 second, retry after >>> 1 second, retry after 1 second" because these will eventually pile up. You >>> want to do something like "retry after 2 seconds + some fuzz factor, retry >>> after 4 seconds plus some fuzz factor, and so forth". >>> >>> - I've been part of a project where there were many administrative >>> "knobs" that we could flip to gracefully degrade and turn off expensive >>> features. This is particularly useful when you push a new version because >>> you can narrow in on the component that seems to be causing your stack to >>> break. It's also nice because you can turn something off, gather with your >>> team and figure out what the plan of attack is without a full rollback. The >>> tradeoff is that you have to build it, and it's non-trivial to have this >>> all over your code, so you need to figure out which parts tend to be more >>> complex than others. In an ideal world, you always know what component does >>> what in your app, but in reality, you usually don't know that there is a >>> weird code path that might invoke the most expensive method call over and >>> over. >>> >>> >>> -- >>> Ikai Lan >>> Developer Programs Engineer, Google App Engine >>> plus.ikailan.com | twitter.com/ikai >>> >>> >>> >>> On Fri, Dec 2, 2011 at 11:31 AM, Ikai Lan (Google) wrote: >>> As far as I know, there are no plans to shut down M/S within the next 12 months. This is NOT an official announcement, so if this does change later (I don't think it will), just know that it's a gut feeling. When there is a plan to official deprecate master/slave, we will announce it. Many new features will be HRD only, however. Python 2.7 is one of them. Full text search *may* be one of them. There are no plans to support Python 2.7 for master/slave applications, and I don't see this changing. I strongly recommend migrating to HRD or at least starting to investigate the possibility. -- Ikai Lan Developer Programs Engineer, Google App Engine plus.ikailan.com | twitter.com/ikai On Fri, Dec 2, 2011 at 9:03 AM, Rishi Arora wrote: > Based on this: > http://code.google.com/appengine/docs/python/datastore/hr/ > > I don't see any strong language indicating imminent deprecation. > Although the fact that python2.7 does not support M/S might hint at that. > If Google does indeed want to deprecate M/S, why wouldn't they just say > it, and give some fixed timeline - 1 month, 6 months, 1 year, whatever. > But why hint it, why not just say it? Since I don't see a strong reason > for
