[google-appengine] Re: Appengine timeout..?
Hi Astrid, On Wed, Jun 17, 2009 at 6:20 AM, astrid.thuec...@googlemail.com < astrid.thuec...@googlemail.com> wrote: > > In general - how much data can I send in an request to the appengine > and be sure it won't time out? Is the amount different between a GET > and POST request? Requests are limited to 10MB, and the counter on your request doesn't start until the entire request has been received. > > > Another thing is, that I would like to create an iframe and send some > data in the src-URL string: > > http://myappspot.appspot.com?somdata=abcdefg"; ... > > > I fear that in special cases there might be many iframes of this type > on a users site and the src-String could be really really long. Does > this mean it would time out? The most data you can reliably transfer in a URL is on the order of about 1k (different browsers have different URL-length limits), which is so small it will take an insignificant amount of time to transmit. > Does it make a difference if there are > many iframes on a single page, since a users connection will try to > load all at once, connection speed is split by the number iframes. > isn't it? Does this increase the danger of running into timeouts? Browsers will request at most two resources from the same site simultaneously, regardless of how many there are to fetch in total. -Nick Johnson --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
Ok, I know browsers have URL length limits. But I was talking about sending data as part of the src="http://myappspot.appspot.com? somdata=abcdefg" of an iframe. So I was wondering are there limits 1. how much data I can stuff in that src-URL? I guess there is no other way to do it if one uses an iframe. 2. are there any time limits while the browser fetches the appengine- webpage for the iframe? Will appengine just time out if the src-URL- data string is too long so it takes to long to finish the request? How long can it take? On Jun 17, 12:41 pm, "Nick Johnson (Google)" wrote: > Hi Astrid, > > On Wed, Jun 17, 2009 at 6:20 AM, astrid.thuec...@googlemail.com < > > astrid.thuec...@googlemail.com> wrote: > > > In general - how much data can I send in an request to the appengine > > and be sure it won't time out? Is the amount different between a GET > > and POST request? > > Requests are limited to 10MB, and the counter on your request doesn't start > until the entire request has been received. > > > > > Another thing is, that I would like to create an iframe and send some > > data in the src-URL string: > > > http://myappspot.appspot.com?somdata=abcdefg"; ... > > > > I fear that in special cases there might be many iframes of this type > > on a users site and the src-String could be really really long. Does > > this mean it would time out? > > The most data you can reliably transfer in a URL is on the order of about 1k > (different browsers have different URL-length limits), which is so small it > will take an insignificant amount of time to transmit. > > > Does it make a difference if there are > > many iframes on a single page, since a users connection will try to > > load all at once, connection speed is split by the number iframes. > > isn't it? Does this increase the danger of running into timeouts? > > Browsers will request at most two resources from the same site > simultaneously, regardless of how many there are to fetch in total. > > -Nick Johnson --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
On Wed, Jun 17, 2009 at 5:58 PM, codingGirl wrote: > > Ok, I know browsers have URL length limits. But I was talking about > sending data as part of the src="http://myappspot.appspot.com? > somdata=abcdefg" of an iframe. So I was wondering are there limits > > 1. how much data I can stuff in that src-URL? I guess there is no > other way to do it if one uses an iframe. That depends on the browser, rather than App Engine. I'm not sure what the hard limits are on what common browsers will accept. > > 2. are there any time limits while the browser fetches the appengine- > webpage for the iframe? Will appengine just time out if the src-URL- > data string is too long so it takes to long to finish the request? How > long can it take? The 30 second request limit only starts after App Engine receives the request. At any rate, with a maximum URL length on the order of a few kilobytes, even the slowest user is not likely to take more than a second or two to transmit the URL! -Nick Johnson > > On Jun 17, 12:41 pm, "Nick Johnson (Google)" > wrote: > > Hi Astrid, > > > > On Wed, Jun 17, 2009 at 6:20 AM, astrid.thuec...@googlemail.com < > > > > astrid.thuec...@googlemail.com> wrote: > > > > > In general - how much data can I send in an request to the appengine > > > and be sure it won't time out? Is the amount different between a GET > > > and POST request? > > > > Requests are limited to 10MB, and the counter on your request doesn't > start > > until the entire request has been received. > > > > > > > > > Another thing is, that I would like to create an iframe and send some > > > data in the src-URL string: > > > > > http://myappspot.appspot.com?somdata=abcdefg"; ... > > > > > > I fear that in special cases there might be many iframes of this type > > > on a users site and the src-String could be really really long. Does > > > this mean it would time out? > > > > The most data you can reliably transfer in a URL is on the order of about > 1k > > (different browsers have different URL-length limits), which is so small > it > > will take an insignificant amount of time to transmit. > > > > > Does it make a difference if there are > > > many iframes on a single page, since a users connection will try to > > > load all at once, connection speed is split by the number iframes. > > > isn't it? Does this increase the danger of running into timeouts? > > > > Browsers will request at most two resources from the same site > > simultaneously, regardless of how many there are to fetch in total. > > > > -Nick Johnson > > > -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
I thought the URL lenght limit applies only for URLs typed int he browsers address bar, doesn't it? Or does it also apply to the src-URL in an iframe? On Jun 17, 7:01 pm, "Nick Johnson (Google)" wrote: > On Wed, Jun 17, 2009 at 5:58 PM, codingGirl > wrote: > > > > > Ok, I know browsers have URL length limits. But I was talking about > > sending data as part of the src="http://myappspot.appspot.com? > > somdata=abcdefg" of an iframe. So I was wondering are there limits > > > 1. how much data I can stuff in that src-URL? I guess there is no > > other way to do it if one uses an iframe. > > That depends on the browser, rather than App Engine. I'm not sure what the > hard limits are on what common browsers will accept. > > > > > 2. are there any time limits while the browser fetches the appengine- > > webpage for the iframe? Will appengine just time out if the src-URL- > > data string is too long so it takes to long to finish the request? How > > long can it take? > > The 30 second request limit only starts after App Engine receives the > request. At any rate, with a maximum URL length on the order of a few > kilobytes, even the slowest user is not likely to take more than a second or > two to transmit the URL! > > -Nick Johnson > > > > > > > On Jun 17, 12:41 pm, "Nick Johnson (Google)" > > wrote: > > > Hi Astrid, > > > > On Wed, Jun 17, 2009 at 6:20 AM, astrid.thuec...@googlemail.com < > > > > astrid.thuec...@googlemail.com> wrote: > > > > > In general - how much data can I send in an request to the appengine > > > > and be sure it won't time out? Is the amount different between a GET > > > > and POST request? > > > > Requests are limited to 10MB, and the counter on your request doesn't > > start > > > until the entire request has been received. > > > > > Another thing is, that I would like to create an iframe and send some > > > > data in the src-URL string: > > > > > http://myappspot.appspot.com?somdata=abcdefg"; ... > > > > > > I fear that in special cases there might be many iframes of this type > > > > on a users site and the src-String could be really really long. Does > > > > this mean it would time out? > > > > The most data you can reliably transfer in a URL is on the order of about > > 1k > > > (different browsers have different URL-length limits), which is so small > > it > > > will take an insignificant amount of time to transmit. > > > > > Does it make a difference if there are > > > > many iframes on a single page, since a users connection will try to > > > > load all at once, connection speed is split by the number iframes. > > > > isn't it? Does this increase the danger of running into timeouts? > > > > Browsers will request at most two resources from the same site > > > simultaneously, regardless of how many there are to fetch in total. > > > > -Nick Johnson > > -- > Nick Johnson, App Engine Developer Programs Engineer > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
2009/6/18 codingGirl > > I thought the URL lenght limit applies only for URLs typed int he > browsers address bar, doesn't it? > > Or does it also apply to the src-URL in an iframe? I think that it is depends on the http server for the maximum length of the request line it can endure. It has to tell from a nomal request to a over-flow attack. Ifram or not is not under the concern, since the server does not care about it is iframe or not. > > > > The 30 second request limit only starts after App Engine receives the > > request. At any rate, with a maximum URL length on the order of a few > > kilobytes, even the slowest user is not likely to take more than a second > or > > two to transmit the URL! > > > I think the "Timeout problem" of the data store is the most imfamouse problem of GAE. It was reported in many usage scenarios. We have encoutered this problem by simply iterate all the members in a small amount of Model. Some others has the same experience. By posted mails, It seems to happen in some unbelivable simple case. I have no idea if the Timeout problem has been improved or not. Definitely, it is the problem which prohibits my boss from investing much more man-hour to get involved in the GAE. It seems that the official answer to this problem is: we do things in wrong way. (Well, what is the right way?) Maybe it has been resolved, or never to be taken as a problem. Who knows the most updated information about that? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
On Thu, Jun 18, 2009 at 10:31 AM, Iap wrote: > > > 2009/6/18 codingGirl > >> >> I thought the URL lenght limit applies only for URLs typed int he >> browsers address bar, doesn't it? >> >> Or does it also apply to the src-URL in an iframe? > > > I think that it is depends on the http server for the maximum length > of the request line it can endure. It has to tell from a nomal request > to a over-flow attack. Ifram or not is not under the concern, since the > server does not care about it is iframe or not. > Both servers and clients have limits on the length of URL they will accept. For example, Internet Explorer is limited to 2048 characters: http://support.microsoft.com/kb/208427 > > >> > >> > The 30 second request limit only starts after App Engine receives the >> > request. At any rate, with a maximum URL length on the order of a few >> > kilobytes, even the slowest user is not likely to take more than a >> second or >> > two to transmit the URL! >> > >> > > I think the "Timeout problem" of the data store is the most imfamouse > problem of GAE. > It was reported in many usage scenarios. We have encoutered this problem > by simply iterate all the members in a small amount of Model. > Some others has the same experience. > By posted mails, It seems to happen in some unbelivable simple case. > I have no idea if the Timeout problem has been improved or not. > Definitely, it is the problem which prohibits my boss from investing > much more man-hour to get involved in the GAE. > It seems that the official answer to this problem is: we do things in wrong > way. > (Well, what is the right way?) > Maybe it has been resolved, or never to be taken as a problem. > Who knows the most updated information about that? > Timeouts can be caused by a number of factors. About 0.1%-0.2% of datastore operations will timeout in normal operation - just due to how Bigtable works, with occasional tablet moves, etcetera. The rate of timeouts can be affected by what you do with the datastore - for example, a single query returning a large number of entities, or for particularly large entities, has a greater chance of timing out than a smaller, but otherwise identical query. For datastore puts and deletes, we currently automatically retry when timeouts occur, and for datastore transactions, your user code is rerun if a timeout occurs. Gets and queries are currently not retried. The solution in general is twofold: Retry when you can, and return an error to the user (or have them re-request the page) in the rare instances when you can't. -Nick Johnson > > > > -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
2009/6/18 Nick Johnson (Google) nick.john...@google.com > > For datastore puts and deletes, we currently automatically retry when > timeouts occur, and for datastore transactions, your user code is rerun if a > timeout occurs. Gets and queries are currently not retried. > It's exciting that the GAE inherites the "I'm feeling lucky" tradition of Google. >_* Why not retry the Get and Queires ? If the retry will eventually succeed, How about provide a decrator for convenience to do the things right. Iap --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
Hi Iap, On Thu, Jun 18, 2009 at 11:01 AM, Iap wrote: > > > 2009/6/18 Nick Johnson (Google) nick.john...@google.com > >> >> For datastore puts and deletes, we currently automatically retry when >> timeouts occur, and for datastore transactions, your user code is rerun if a >> timeout occurs. Gets and queries are currently not retried. >> > > It's exciting that the GAE inherites the "I'm feeling lucky" tradition of > Google. >_* > > Why not retry the Get and Queires ? > We are planning to add that in a future release. If the retry will eventually succeed, > How about provide a decrator for convenience to do the things right. > Retries won't _always_ succeed, but they will avoid transient timeouts, reducing the number of user-visible timeout errors. -Nick Johnson > Iap > > > > > -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
2009/6/18 Nick Johnson (Google) Hi, Nick >> Why not retry the Get and Queires ? > > We are planning to add that in a future release. That would be great! I have good reason to persuade my boss to re-open her mind. Iap --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
I asked about this (datastore timeout exceptions) in yesterday's app engine chat and just wanted to follow up. I'm trying to figure out an architecture to deal with the timeout exceptions. The "solution" of pushing the exception back to the end user is actually ok for my app. This works for read timeouts. For write timeouts, my code can be in the middle of storing a data structure when the timeout happens. Thus, I could have a corrupt data structure instance in my datastore. So, for fault-tolerant production code, any (later) access to any datastore data structure should check the integrity data structure before using it. This is what Ken Ashcraft's google i/o 2008 talk recommended (without referring specifically to timeout issues): http://sites.google.com/site/io/best-practices---building-a-production-quality-application-on-google-app-engine A bit troublesome but at least we know what to do: every datastore read needs to do data structure integrity checking. I just want to get feedback on whether this line of thinking and the resulting coding style is what the appengine team recommends. Also, it would be very helpful to know what google engineers do to avoid this issue (do they use the same style or do they have internal tools or libraries that hide this issue from them). Dennis search term: DatastoreTimeoutException On Jun 18, 6:01 pm, Iap wrote: > 2009/6/18 Nick Johnson (Google) nick.john...@google.com > > > > > For datastore puts and deletes, we currently automatically retry when > > timeouts occur, and for datastore transactions, your user code is rerun if a > > timeout occurs. Gets and queries are currently not retried. > > It's exciting that the GAE inherites the "I'm feeling lucky" tradition of > Google. >_* > > Why not retry the Get and Queires ? > If the retry will eventually succeed, > How about provide a decrator for convenience to do the things right. > > Iap --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
On Thu, Jun 18, 2009 at 11:45 AM, Dennis wrote: > > I asked about this (datastore timeout exceptions) in yesterday's app > engine chat and just wanted to follow up. > > I'm trying to figure out an architecture to deal with the timeout > exceptions. > The "solution" of pushing the exception back to the end user is > actually ok for my app. > This works for read timeouts. > > For write timeouts, my code can be in the middle of storing a data > structure when the timeout happens. > Thus, I could have a corrupt data structure instance in my datastore. I presume by 'data structure', you mean multiple entities? Because individual entities (and entity groups inside transactions) will always be written atomically - all or nothing - and never 'corrupted' or partially written. > So, for fault-tolerant production code, any (later) access to any > datastore data structure should check > the integrity data structure before using it. > This is what Ken Ashcraft's google i/o 2008 talk recommended (without > referring specifically to timeout issues): > > http://sites.google.com/site/io/best-practices---building-a-production-quality-application-on-google-app-engine > > A bit troublesome but at least we know what to do: > every datastore read needs to do data structure integrity checking. > > > I just want to get feedback on whether this line of thinking and the > resulting coding style is what the appengine team recommends. > Also, it would be very helpful to know what google engineers do to > avoid this issue (do they use the same style or do they have internal > tools or libraries that hide this issue from them). More or less. If your write order is predictable, some of your queries will probably not care about incomplete data, or that they may see data that's only partially written. In general, though, the 'roll forward' approach works quite well. -Nick Johnson > > > Dennis > > search term: DatastoreTimeoutException > > > > On Jun 18, 6:01 pm, Iap wrote: > > 2009/6/18 Nick Johnson (Google) nick.john...@google.com > > > > > > > > > For datastore puts and deletes, we currently automatically retry when > > > timeouts occur, and for datastore transactions, your user code is rerun > if a > > > timeout occurs. Gets and queries are currently not retried. > > > > It's exciting that the GAE inherites the "I'm feeling lucky" tradition of > > Google. >_* > > > > Why not retry the Get and Queires ? > > If the retry will eventually succeed, > > How about provide a decrator for convenience to do the things right. > > > > Iap > > > -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
> I presume by 'data structure', you mean multiple entities? Yes, multiple entity structures need to be integrity checked each time they are used. Thanks for the confirmation of how we should code. > only partially written. In general, though, the 'roll forward' approach > works quite well. What is the 'roll forward' approach? On Jun 18, 6:49 pm, "Nick Johnson (Google)" wrote: > On Thu, Jun 18, 2009 at 11:45 AM, Dennis wrote: > > > I asked about this (datastore timeout exceptions) in yesterday's app > > engine chat and just wanted to follow up. > > > I'm trying to figure out an architecture to deal with the timeout > > exceptions. > > The "solution" of pushing the exception back to the end user is > > actually ok for my app. > > This works for read timeouts. > > > For write timeouts, my code can be in the middle of storing a data > > structure when the timeout happens. > > Thus, I could have a corrupt data structure instance in my datastore. > > I presume by 'data structure', you mean multiple entities? Because > individual entities (and entity groups inside transactions) will always be > written atomically - all or nothing - and never 'corrupted' or partially > written. > > > > > > > So, for fault-tolerant production code, any (later) access to any > > datastore data structure should check > > the integrity data structure before using it. > > This is what Ken Ashcraft's google i/o 2008 talk recommended (without > > referring specifically to timeout issues): > > >http://sites.google.com/site/io/best-practices---building-a-productio... > > > A bit troublesome but at least we know what to do: > > every datastore read needs to do data structure integrity checking. > > > I just want to get feedback on whether this line of thinking and the > > resulting coding style is what the appengine team recommends. > > Also, it would be very helpful to know what google engineers do to > > avoid this issue (do they use the same style or do they have internal > > tools or libraries that hide this issue from them). > > More or less. If your write order is predictable, some of your queries will > probably not care about incomplete data, or that they may see data that's > only partially written. In general, though, the 'roll forward' approach > works quite well. > > -Nick Johnson > > > > > > > > > Dennis > > > search term: DatastoreTimeoutException > > > On Jun 18, 6:01 pm, Iap wrote: > > > 2009/6/18 Nick Johnson (Google) nick.john...@google.com > > > > > For datastore puts and deletes, we currently automatically retry when > > > > timeouts occur, and for datastore transactions, your user code is rerun > > if a > > > > timeout occurs. Gets and queries are currently not retried. > > > > It's exciting that the GAE inherites the "I'm feeling lucky" tradition of > > > Google. >_* > > > > Why not retry the Get and Queires ? > > > If the retry will eventually succeed, > > > How about provide a decrator for convenience to do the things right. > > > > Iap > > -- > Nick Johnson, App Engine Developer Programs Engineer > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > 368047- Hide quoted text - > > - Show quoted text -- Hide quoted text - > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
PS: what does google do internally (wrt coding to avoid datastore timeouts)? On Jun 18, 6:58 pm, Dennis wrote: > > I presume by 'data structure', you mean multiple entities? > > Yes, multiple entity structures need to be integrity checked each time > they are used. > Thanks for the confirmation of how we should code. > > > only partially written. In general, though, the 'roll forward' approach > > works quite well. > > What is the 'roll forward' approach? > > On Jun 18, 6:49 pm, "Nick Johnson (Google)" > wrote: > > > > > On Thu, Jun 18, 2009 at 11:45 AM, Dennis wrote: > > > > I asked about this (datastore timeout exceptions) in yesterday's app > > > engine chat and just wanted to follow up. > > > > I'm trying to figure out an architecture to deal with the timeout > > > exceptions. > > > The "solution" of pushing the exception back to the end user is > > > actually ok for my app. > > > This works for read timeouts. > > > > For write timeouts, my code can be in the middle of storing a data > > > structure when the timeout happens. > > > Thus, I could have a corrupt data structure instance in my datastore. > > > I presume by 'data structure', you mean multiple entities? Because > > individual entities (and entity groups inside transactions) will always be > > written atomically - all or nothing - and never 'corrupted' or partially > > written. > > > > So, for fault-tolerant production code, any (later) access to any > > > datastore data structure should check > > > the integrity data structure before using it. > > > This is what Ken Ashcraft's google i/o 2008 talk recommended (without > > > referring specifically to timeout issues): > > > >http://sites.google.com/site/io/best-practices---building-a-productio... > > > > A bit troublesome but at least we know what to do: > > > every datastore read needs to do data structure integrity checking. > > > > I just want to get feedback on whether this line of thinking and the > > > resulting coding style is what the appengine team recommends. > > > Also, it would be very helpful to know what google engineers do to > > > avoid this issue (do they use the same style or do they have internal > > > tools or libraries that hide this issue from them). > > > More or less. If your write order is predictable, some of your queries will > > probably not care about incomplete data, or that they may see data that's > > only partially written. In general, though, the 'roll forward' approach > > works quite well. > > > -Nick Johnson > > > > Dennis > > > > search term: DatastoreTimeoutException > > > > On Jun 18, 6:01 pm, Iap wrote: > > > > 2009/6/18 Nick Johnson (Google) nick.john...@google.com > > > > > > For datastore puts and deletes, we currently automatically retry when > > > > > timeouts occur, and for datastore transactions, your user code is > > > > > rerun > > > if a > > > > > timeout occurs. Gets and queries are currently not retried. > > > > > It's exciting that the GAE inherites the "I'm feeling lucky" tradition > > > > of > > > > Google. >_* > > > > > Why not retry the Get and Queires ? > > > > If the retry will eventually succeed, > > > > How about provide a decrator for convenience to do the things right. > > > > > Iap > > > -- > > Nick Johnson, App Engine Developer Programs Engineer > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > > 368047- Hide quoted text - > > > - Show quoted text -- Hide quoted text - > > > - Show quoted text -- Hide quoted text - > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Appengine timeout..?
On Thu, Jun 18, 2009 at 12:00 PM, Dennis wrote:
>
> PS: what does google do internally (wrt coding to avoid datastore
> timeouts)?
The same as we recommend to everyone else. :)
>
> On Jun 18, 6:58 pm, Dennis wrote:
> > > I presume by 'data structure', you mean multiple entities?
> >
> > Yes, multiple entity structures need to be integrity checked each time
> > they are used.
> > Thanks for the confirmation of how we should code.
> >
> > > only partially written. In general, though, the 'roll forward' approach
> > > works quite well.
> >
> > What is the 'roll forward' approach?
>
The approach documented in the talk you referenced: First, record your
intent. Then, execute it, and if your transaction fails, a later read can
detect this and pick up where it left off ('rolling forward' the
transaction).
-Nick Johnson
> >
> > On Jun 18, 6:49 pm, "Nick Johnson (Google)"
> > wrote:
> >
> >
> >
> > > On Thu, Jun 18, 2009 at 11:45 AM, Dennis wrote:
> >
> > > > I asked about this (datastore timeout exceptions) in yesterday's app
> > > > engine chat and just wanted to follow up.
> >
> > > > I'm trying to figure out an architecture to deal with the timeout
> > > > exceptions.
> > > > The "solution" of pushing the exception back to the end user is
> > > > actually ok for my app.
> > > > This works for read timeouts.
> >
> > > > For write timeouts, my code can be in the middle of storing a data
> > > > structure when the timeout happens.
> > > > Thus, I could have a corrupt data structure instance in my datastore.
> >
> > > I presume by 'data structure', you mean multiple entities? Because
> > > individual entities (and entity groups inside transactions) will always
> be
> > > written atomically - all or nothing - and never 'corrupted' or
> partially
> > > written.
> >
> > > > So, for fault-tolerant production code, any (later) access to any
> > > > datastore data structure should check
> > > > the integrity data structure before using it.
> > > > This is what Ken Ashcraft's google i/o 2008 talk recommended (without
> > > > referring specifically to timeout issues):
> >
> > > >
> http://sites.google.com/site/io/best-practices---building-a-productio...
> >
> > > > A bit troublesome but at least we know what to do:
> > > > every datastore read needs to do data structure integrity checking.
> >
> > > > I just want to get feedback on whether this line of thinking and the
> > > > resulting coding style is what the appengine team recommends.
> > > > Also, it would be very helpful to know what google engineers do to
> > > > avoid this issue (do they use the same style or do they have internal
> > > > tools or libraries that hide this issue from them).
> >
> > > More or less. If your write order is predictable, some of your queries
> will
> > > probably not care about incomplete data, or that they may see data
> that's
> > > only partially written. In general, though, the 'roll forward' approach
> > > works quite well.
> >
> > > -Nick Johnson
> >
> > > > Dennis
> >
> > > > search term: DatastoreTimeoutException
> >
> > > > On Jun 18, 6:01 pm, Iap wrote:
> > > > > 2009/6/18 Nick Johnson (Google) nick.john...@google.com
> >
> > > > > > For datastore puts and deletes, we currently automatically retry
> when
> > > > > > timeouts occur, and for datastore transactions, your user code is
> rerun
> > > > if a
> > > > > > timeout occurs. Gets and queries are currently not retried.
> >
> > > > > It's exciting that the GAE inherites the "I'm feeling lucky"
> tradition of
> > > > > Google. >_*
> >
> > > > > Why not retry the Get and Queires ?
> > > > > If the retry will eventually succeed,
> > > > > How about provide a decrator for convenience to do the things
> right.
> >
> > > > > Iap
> >
> > > --
> > > Nick Johnson, App Engine Developer Programs Engineer
> > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration
> Number:
> > > 368047- Hide quoted text -
> >
> > > - Show quoted text -- Hide quoted text -
> >
> > > - Show quoted text -- Hide quoted text -
> >
> > - Show quoted text -
> >
>
--
Nick Johnson, App Engine Developer Programs Engineer
Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number:
368047
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
