[google-appengine] Re: Lots of user-facing loading requests
Hi Francois, Don't set min idles to automatic, set it to a high enough number that over time Total instances (blue) == Billed instances (green). Leave max idles and the two latency settings at automatic. If you do have min idles configured, can you post a screencap of your instances graph? /Tom On Jan 18, 8:45 am, Francois Masurel f.masu...@gmail.com wrote: Things are getting worse : 20 loading requests for the last 23 minutes (about 20% of all requests). Could I have an explanation ? What am I doing wrong ? I'm really lost :-( App Id : vncts1 On Friday, January 18, 2013 2:25:17 PM UTC+1, Francois Masurel wrote: I have been testing full automatic mode for the last hours (all application settings set to automatic). Still getting quite a few user-facing loading requests (~20 per hour, 5% of all requests), there is no more warmup requests in the logs. One dynamic instance has been alive since the beginning of the test and is getting most of the traffic. Are these numbers normal ? François App Id: vncts1 On Thursday, January 17, 2013 11:21:16 PM UTC+1, Francois Masurel wrote: Hi Tom, Thanx for your suggestion, that's just the settings I'm testing at the moment. Seems to reduce significantly user-facing requests but will quite increase my bill (x4) :-( May be it will be worth it. I'll tell you how it goes. François On Thursday, January 17, 2013 11:11:56 PM UTC+1, Tom Phillips wrote: Make sure on your instances graph that the blue Total line is not often, ideally only under unforeseen bursts, going higher than the green Billed line. Total(blue) Billed(green): you aren't charged for the delta between Total and Billed, but lower QOS for users (assuming your startup time is high) since they get most loading requests. Total(blue) == Billed(green): Your /_ah/warmup gets all loading requests, optimal QOS This is just what I've observed (leaving max idles and min/max latency at automatic). Unless you have magically low startup latency, boost min idles until Total == Billed consistently. /Tom On Jan 17, 11:56 am, Francois Masurel f.masu...@gmail.com wrote: For my low traffic website I have found that 50 new instances were started during the last 2 hours and 22 minutes (taken from logs searching for new process). Seems quite a lot for me as I already have a resident instance. But what annoys me the most is that, among those 50 loading requests, 20 were user-facing requests (40%) and 5 were cron requests (10%). Is it normal behavior ? Thanx for your help. François -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Lots of user-facing loading requests
Yep, you need more min idles. At some min idle setting you'll see that the blue line no longer goes above green (minus unforeseen bursts, which need be you can prepare for with an even higher min idle setting). Once blue doesn't go above green, you'll see warmup request going to / _ah/warmup and not to your users. Basically, you don't get any delta between blue and green for free. You pay for it by having your users suffer loading requests. That's one reason why very low startup latency apps shine - they can take advantage of any delta. Higher startup latency apps (pretty much any real java app) are OK at steady state, but only if you are willing to pay to have a high enough min idle setting that you are billed for all running instances. /Tom On Jan 18, 12:07 pm, Francois Masurel f.masu...@gmail.com wrote: Here is my dashboard instance graph. First part is with 1 min idle instance, second part with min idle instance set to auto. https://lh6.googleusercontent.com/-xVIJBwSd04o/UPmA5FfU4QI/z5... I have just set min idle instances to 2 and see how it goes. François On Friday, January 18, 2013 5:11:47 PM UTC+1, Tom Phillips wrote: Hi Francois, Don't set min idles to automatic, set it to a high enough number that over time Total instances (blue) == Billed instances (green). Leave max idles and the two latency settings at automatic. If you do have min idles configured, can you post a screencap of your instances graph? /Tom On Jan 18, 8:45 am, Francois Masurel f.masu...@gmail.com wrote: Things are getting worse : 20 loading requests for the last 23 minutes (about 20% of all requests). Could I have an explanation ? What am I doing wrong ? I'm really lost :-( App Id : vncts1 On Friday, January 18, 2013 2:25:17 PM UTC+1, Francois Masurel wrote: I have been testing full automatic mode for the last hours (all application settings set to automatic). Still getting quite a few user-facing loading requests (~20 per hour, 5% of all requests), there is no more warmup requests in the logs. One dynamic instance has been alive since the beginning of the test and is getting most of the traffic. Are these numbers normal ? François App Id: vncts1 On Thursday, January 17, 2013 11:21:16 PM UTC+1, Francois Masurel wrote: Hi Tom, Thanx for your suggestion, that's just the settings I'm testing at the moment. Seems to reduce significantly user-facing requests but will quite increase my bill (x4) :-( May be it will be worth it. I'll tell you how it goes. François On Thursday, January 17, 2013 11:11:56 PM UTC+1, Tom Phillips wrote: Make sure on your instances graph that the blue Total line is not often, ideally only under unforeseen bursts, going higher than the green Billed line. Total(blue) Billed(green): you aren't charged for the delta between Total and Billed, but lower QOS for users (assuming your startup time is high) since they get most loading requests. Total(blue) == Billed(green): Your /_ah/warmup gets all loading requests, optimal QOS This is just what I've observed (leaving max idles and min/max latency at automatic). Unless you have magically low startup latency, boost min idles until Total == Billed consistently. /Tom On Jan 17, 11:56 am, Francois Masurel f.masu...@gmail.com wrote: For my low traffic website I have found that 50 new instances were started during the last 2 hours and 22 minutes (taken from logs searching for new process). Seems quite a lot for me as I already have a resident instance. But what annoys me the most is that, among those 50 loading requests, 20 were user-facing requests (40%) and 5 were cron requests (10%). Is it normal behavior ? Thanx for your help. François -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Lots of user-facing loading requests
Hi Carl, What does your instances graph look like? Can you provide a screencap? What I'm wondering is if for extremely low traffic apps there is no way to keep Total instances from sometimes exceeding Billed instances (and hence giving users loading requests). /Tom On Jan 18, 1:21 pm, Carl Schroeder schroeder.car...@gmail.com wrote: 2 Resident instances. App was quiet for 10 minutes, no requests being served. 1 request sent. Neither resident instance took it. A cold instance was started without using /ah_/warmup. That user facing request languished for 30 seconds while GAE java spun up. There seems to be no way to prevent these sorts of catastrophes from happening. App ID = lemurspot Instance ID = 00c61b117cb19763aeea814cbc78e80de2f437 On Friday, January 18, 2013 10:10:25 AM UTC-8, Carl Schroeder wrote: There needs to be a checkbox on the dashboard which says never send user request to cold instances. Then low traffic apps (or versions) with low latency handlers will no longer be at the mercy of a load balancing algorithm making WAGs in an information vacuum. After an app (or version) has enough traffic for the instance scheduler to make educated guesses, this box can be unchecked. I think that would fix everyone's issues with cold start latency on GAE java. On Friday, January 18, 2013 9:22:15 AM UTC-8, Tom Phillips wrote: Yep, you need more min idles. At some min idle setting you'll see that the blue line no longer goes above green (minus unforeseen bursts, which need be you can prepare for with an even higher min idle setting). Once blue doesn't go above green, you'll see warmup request going to / _ah/warmup and not to your users. Basically, you don't get any delta between blue and green for free. You pay for it by having your users suffer loading requests. That's one reason why very low startup latency apps shine - they can take advantage of any delta. Higher startup latency apps (pretty much any real java app) are OK at steady state, but only if you are willing to pay to have a high enough min idle setting that you are billed for all running instances. /Tom On Jan 18, 12:07 pm, Francois Masurel f.masu...@gmail.com wrote: Here is my dashboard instance graph. First part is with 1 min idle instance, second part with min idle instance set to auto. https://lh6.googleusercontent.com/-xVIJBwSd04o/UPmA5FfU4QI/z5... I have just set min idle instances to 2 and see how it goes. François On Friday, January 18, 2013 5:11:47 PM UTC+1, Tom Phillips wrote: Hi Francois, Don't set min idles to automatic, set it to a high enough number that over time Total instances (blue) == Billed instances (green). Leave max idles and the two latency settings at automatic. If you do have min idles configured, can you post a screencap of your instances graph? /Tom On Jan 18, 8:45 am, Francois Masurel f.masu...@gmail.com wrote: Things are getting worse : 20 loading requests for the last 23 minutes (about 20% of all requests). Could I have an explanation ? What am I doing wrong ? I'm really lost :-( App Id : vncts1 On Friday, January 18, 2013 2:25:17 PM UTC+1, Francois Masurel wrote: I have been testing full automatic mode for the last hours (all application settings set to automatic). Still getting quite a few user-facing loading requests (~20 per hour, 5% of all requests), there is no more warmup requests in the logs. One dynamic instance has been alive since the beginning of the test and is getting most of the traffic. Are these numbers normal ? François App Id: vncts1 On Thursday, January 17, 2013 11:21:16 PM UTC+1, Francois Masurel wrote: Hi Tom, Thanx for your suggestion, that's just the settings I'm testing at the moment. Seems to reduce significantly user-facing requests but will quite increase my bill (x4) :-( May be it will be worth it. I'll tell you how it goes. François On Thursday, January 17, 2013 11:11:56 PM UTC+1, Tom Phillips wrote: Make sure on your instances graph that the blue Total line is not often, ideally only under unforeseen bursts, going higher than the green Billed line. Total(blue) Billed(green): you aren't charged for the delta between Total and Billed, but lower QOS for users (assuming your startup time is high) since they get most loading requests. Total(blue) == Billed(green): Your /_ah/warmup gets all loading requests, optimal QOS This is just what I've observed (leaving max idles and min/max latency at automatic). Unless you have magically low startup latency, boost min idles until Total == Billed consistently. /Tom On Jan 17, 11:56 am, Francois Masurel f.masu
[google-appengine] Re: Lots of user-facing loading requests
You are still getting Total(blue)Billed(green) so will see user facing loads rather than /_ah/warmup. Just to clarify, you have the following settings: Min Idle: 2 (for most of the graph couldn't have been that because Billed==1, but I'm assuming you changed it recently to 2) Max Idle: Automatic Min/Max Latency: Automatic If so, try changing Min Idle to 4, or higher to see if you can get the Total (blue) line to stop jumping over the Billed (green) line. i.e. pay for all instances that are running. The more latency in the app, including startup latency, the higher min idle setting you'll need to achieve this for a given traffic pattern. It's TBD to see you can achieve steady Total==Billed, but if you can you should stop seeing user facing loads (albeit with increased operational cost for your production app). /T On Jan 18, 1:56 pm, Carl Schroeder schroeder.car...@gmail.com wrote: Attached is my instance graph for the past 24 hours. My app gets the sort of traffic that one instance can satisfy effortlessly (but I have no way to configure that). I am running 2 resident instances (which should be monumental overkill) and still getting cold starts. FYI, this app can run on 1 micro AWS instance. On Friday, January 18, 2013 10:29:02 AM UTC-8, Tom Phillips wrote: Hi Carl, What does your instances graph look like? Can you provide a screencap? What I'm wondering is if for extremely low traffic apps there is no way to keep Total instances from sometimes exceeding Billed instances (and hence giving users loading requests). /Tom On Jan 18, 1:21 pm, Carl Schroeder schroeder.car...@gmail.com wrote: 2 Resident instances. App was quiet for 10 minutes, no requests being served. 1 request sent. Neither resident instance took it. A cold instance was started without using /ah_/warmup. That user facing request languished for 30 seconds while GAE java spun up. There seems to be no way to prevent these sorts of catastrophes from happening. App ID = lemurspot Instance ID = 00c61b117cb19763aeea814cbc78e80de2f437 On Friday, January 18, 2013 10:10:25 AM UTC-8, Carl Schroeder wrote: There needs to be a checkbox on the dashboard which says never send user request to cold instances. Then low traffic apps (or versions) with low latency handlers will no longer be at the mercy of a load balancing algorithm making WAGs in an information vacuum. After an app (or version) has enough traffic for the instance scheduler to make educated guesses, this box can be unchecked. I think that would fix everyone's issues with cold start latency on GAE java. On Friday, January 18, 2013 9:22:15 AM UTC-8, Tom Phillips wrote: Yep, you need more min idles. At some min idle setting you'll see that the blue line no longer goes above green (minus unforeseen bursts, which need be you can prepare for with an even higher min idle setting). Once blue doesn't go above green, you'll see warmup request going to / _ah/warmup and not to your users. Basically, you don't get any delta between blue and green for free. You pay for it by having your users suffer loading requests. That's one reason why very low startup latency apps shine - they can take advantage of any delta. Higher startup latency apps (pretty much any real java app) are OK at steady state, but only if you are willing to pay to have a high enough min idle setting that you are billed for all running instances. /Tom On Jan 18, 12:07 pm, Francois Masurel f.masu...@gmail.com wrote: Here is my dashboard instance graph. First part is with 1 min idle instance, second part with min idle instance set to auto. https://lh6.googleusercontent.com/-xVIJBwSd04o/UPmA5FfU4QI/z5... I have just set min idle instances to 2 and see how it goes. François On Friday, January 18, 2013 5:11:47 PM UTC+1, Tom Phillips wrote: Hi Francois, Don't set min idles to automatic, set it to a high enough number that over time Total instances (blue) == Billed instances (green). Leave max idles and the two latency settings at automatic. If you do have min idles configured, can you post a screencap of your instances graph? /Tom On Jan 18, 8:45 am, Francois Masurel f.masu...@gmail.com wrote: Things are getting worse : 20 loading requests for the last 23 minutes (about 20% of all requests). Could I have an explanation ? What am I doing wrong ? I'm really lost :-( App Id : vncts1 On Friday, January 18, 2013 2:25:17 PM UTC+1, Francois Masurel wrote: I have been testing full automatic mode for the last hours (all application settings set to automatic). Still getting quite a few user-facing loading requests (~20 per hour, 5% of all requests
[google-appengine] Re: Lots of user-facing loading requests
Make sure on your instances graph that the blue Total line is not often, ideally only under unforeseen bursts, going higher than the green Billed line. Total(blue) Billed(green): you aren't charged for the delta between Total and Billed, but lower QOS for users (assuming your startup time is high) since they get most loading requests. Total(blue) == Billed(green): Your /_ah/warmup gets all loading requests, optimal QOS This is just what I've observed (leaving max idles and min/max latency at automatic). Unless you have magically low startup latency, boost min idles until Total == Billed consistently. /Tom On Jan 17, 11:56 am, Francois Masurel f.masu...@gmail.com wrote: For my low traffic website I have found that 50 new instances were started during the last 2 hours and 22 minutes (taken from logs searching for new process). Seems quite a lot for me as I already have a resident instance. But what annoys me the most is that, among those 50 loading requests, 20 were user-facing requests (40%) and 5 were cron requests (10%). Is it normal behavior ? Thanx for your help. François -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Startup time exceeded...on F4?!
Hi Takashi, Most of my traffic is from other systems, with predictable ramp ups/ down at peak periods during week days (app is cliniconexapp-hrd). Not sure if it would be considered spiky. Right now I have 1 resident and 4 dynamic. That's more instances than usual though - usually a couple only, and App Engine just seems to be keeping a few extra around right now for whatever reason. Java, multi-threaded. Max idle instances is automatic. The resident instance handles almost no requests (only 3 so far). If a dynamic instance needs to be started, couldn't the scheduler send at least some requests to the idle instance while calling /_ah/warmup on the spawning dynamic one? The main problem with loading requests for me is for an incoming SMS handler for Twilio. Twilio callbacks have a hard timeout of 15 seconds, so if it hits a loading request the user will see a fallback error SMS response. Not ideal, and i'd like to minimize the chances of it. It sounds like I could pay to reserve even more excess capacity to do this, but I'd think that my 1 resident instance could at least be used a bit (say 30% load temporarily) to allow some predictive loading of dynamic instances via /_ah/warmup. /Tom On Jul 14, 9:49 am, Takashi Matsuo tmat...@google.com wrote: On Jul 14, 2012 12:42 PM, Tom Phillips tphill0...@gmail.com wrote: Hi Takashi, Now that I've added a resident idle instance, the logs seem to confirm that only the loading of the resident instance is via /_ah/warmup. It's difficult to tell that for sure, but all my current dynamic instances were loaded on a user URL, and the current resident instance was loaded via /_ah/warmup. Yes, it is expected behavior. Resident idle instances are loaded only infrequently, and serve little traffic, so of what use are warmup requests if only they get them? The vast majority of loading requests are now on user URLs. That's a 20+ second wait (Java) by a user on almost every loading request. Those resident instances are used when there is no available dynamic instance. If there's no available resident instance, your user will see the loading requests. In general, if you see too many user-facing loading requests, there is still a room for raising your number of resident instances in order to reduce the number of user-facing loading requests. Back in the Always-on days, I would see virtually all loading requests use /_ah/warmup, as long as traffic was relatively stable. So an instance loading rarely affected an actual user. What do we configure now to get this same behavior? Probably you can not expect the very same behavior. How many resident instances are you configured? Are you configured the 'Max Idle instances' to 'Automatic'? How spiky is your access pattern? Are you using the concurrent requests? -- Takashi /Tom On Jul 13, 7:07 pm, Takashi Matsuo tmat...@google.com wrote: Hi Mauricio, Yes I understand that, and we've been working on it. Sorry that it's taking long. Hi Wilson, It's coming with the pricing change. Since we started charging with number of instances, we needed to provide more control on how we spin-up a new instances. In other words, if we continued the old behavior, many developers would have complained like App Engine spins up unnecessary instances.. We believe that it makes more sense when warmup requests are used only for resident idle instances. -- Takashi On Sat, Jul 14, 2012 at 7:18 AM, Wilson MacGyver wmacgy...@gmail.com wrote: any reason behind this change? this seems kind of strange. I too was wondering by the behavior till you posted this. I had my min set to automatic also On Fri, Jul 13, 2012 at 9:28 AM, Takashi Matsuo tmat...@google.com wrote: Now the warmup requests are fired only if you set min idle instances to some value(not automatic). -- Takashi On Fri, Jul 13, 2012 at 9:52 PM, Michael Hermus michael.her...@gmail.com wrote: Same for me.. I just checked: no calls to warmup, lots of loading requests. **shakes fist at App Engine** On Thursday, July 12, 2012 4:39:48 PM UTC-4, Tom Phillips wrote: Interesting..I checked and I too have 100% of my loading requests on user facing URLs instead of /_ah/warmup. Warmup requests are enabled and Automatic-Automatic for both instance sliders. I used to see at least a decent percentage of loading requests on /_ah/ warmup, but haven't looked in quite a while. /Tom On Jul 12, 3:46 pm, David Hardwick david.hardw...@bettercloud.com wrote: Some additional observations and questions... After reading this [Link 1] stack overflow article that mentioned an issue with having your Max Idle count below 6, we started looking at our warmup request on our staging environment because that app-id has Idle Instances set to Auto
[google-appengine] Re: Startup time exceeded...on F4?!
Hi Takashi, Now that I've added a resident idle instance, the logs seem to confirm that only the loading of the resident instance is via /_ah/warmup. It's difficult to tell that for sure, but all my current dynamic instances were loaded on a user URL, and the current resident instance was loaded via /_ah/warmup. Resident idle instances are loaded only infrequently, and serve little traffic, so of what use are warmup requests if only they get them? The vast majority of loading requests are now on user URLs. That's a 20+ second wait (Java) by a user on almost every loading request. Back in the Always-on days, I would see virtually all loading requests use /_ah/warmup, as long as traffic was relatively stable. So an instance loading rarely affected an actual user. What do we configure now to get this same behavior? /Tom On Jul 13, 7:07 pm, Takashi Matsuo tmat...@google.com wrote: Hi Mauricio, Yes I understand that, and we've been working on it. Sorry that it's taking long. Hi Wilson, It's coming with the pricing change. Since we started charging with number of instances, we needed to provide more control on how we spin-up a new instances. In other words, if we continued the old behavior, many developers would have complained like App Engine spins up unnecessary instances.. We believe that it makes more sense when warmup requests are used only for resident idle instances. -- Takashi On Sat, Jul 14, 2012 at 7:18 AM, Wilson MacGyver wmacgy...@gmail.comwrote: any reason behind this change? this seems kind of strange. I too was wondering by the behavior till you posted this. I had my min set to automatic also On Fri, Jul 13, 2012 at 9:28 AM, Takashi Matsuo tmat...@google.com wrote: Now the warmup requests are fired only if you set min idle instances to some value(not automatic). -- Takashi On Fri, Jul 13, 2012 at 9:52 PM, Michael Hermus michael.her...@gmail.com wrote: Same for me.. I just checked: no calls to warmup, lots of loading requests. **shakes fist at App Engine** On Thursday, July 12, 2012 4:39:48 PM UTC-4, Tom Phillips wrote: Interesting..I checked and I too have 100% of my loading requests on user facing URLs instead of /_ah/warmup. Warmup requests are enabled and Automatic-Automatic for both instance sliders. I used to see at least a decent percentage of loading requests on /_ah/ warmup, but haven't looked in quite a while. /Tom On Jul 12, 3:46 pm, David Hardwick david.hardw...@bettercloud.com wrote: Some additional observations and questions... After reading this [Link 1] stack overflow article that mentioned an issue with having your Max Idle count below 6, we started looking at our warmup request on our staging environment because that app-id has Idle Instances set to Auto-Auto, while production had specific values. But...Where did all the /_ah/warmup requests go? When doing a label search for these staging environment logs [path:/_ah/warmup (doing a label search)] we couldn't find any warmup request!!(yes, we have warmup requests turned on)...we would just see the first cold-start request would take around 15 seconds to load (F1) and 10 seconds to load on (F2). I even shut down every instance and hit the staging server again to see if I could find a warmup request in the logs...nope. Honestly, I would rather have a user wait 10 seconds for the first request to that server as opposed risking the warmup requests failing again. Where did all the /_ah/warmup requests go? More importantly, why would we have such different times for warmup requests compared to cold starts? Shouldn't they be nearly identical?! Rock on, -Hardwick [Link 1] - http://stackoverflow.com/questions/9422698/ah-warmup-producing-hardde... On Jul 12, 12:26 pm, David Hardwick david.hardw...@bettercloud.com wrote: Hello, I realize there's been a lot of discussion on startup times exceeded on this forum recently, but wanted needed to post this experience we had this morning to keep the attention on this important issue. We uploaded a point release of our app to a not-live version this morning and, of course, we were going to click around on that instance to make sure it's all kosher before making that version live. The warm-up requests for the not-live version were exceeding the deadline limit of 60s... __and__we__are__on__F4s__!_!. However, the LIVE version of the app crashed too, 500 server errors, instance counts went to zero, all sorts of whacky stuff was seen in the control panel. All that happened to our LIVE version without when all we did was upload another non-live version and hit it with a single request...did I mention we were on F4s? ;-) Does the failure of any
[google-appengine] Re: Startup time exceeded...on F4?!
Interesting..I checked and I too have 100% of my loading requests on user facing URLs instead of /_ah/warmup. Warmup requests are enabled and Automatic-Automatic for both instance sliders. I used to see at least a decent percentage of loading requests on /_ah/ warmup, but haven't looked in quite a while. /Tom On Jul 12, 3:46 pm, David Hardwick david.hardw...@bettercloud.com wrote: Some additional observations and questions... After reading this [Link 1] stack overflow article that mentioned an issue with having your Max Idle count below 6, we started looking at our warmup request on our staging environment because that app-id has Idle Instances set to Auto-Auto, while production had specific values. But...Where did all the /_ah/warmup requests go? When doing a label search for these staging environment logs [path:/_ah/warmup (doing a label search)] we couldn't find any warmup request!!(yes, we have warmup requests turned on)...we would just see the first cold-start request would take around 15 seconds to load (F1) and 10 seconds to load on (F2). I even shut down every instance and hit the staging server again to see if I could find a warmup request in the logs...nope. Honestly, I would rather have a user wait 10 seconds for the first request to that server as opposed risking the warmup requests failing again. Where did all the /_ah/warmup requests go? More importantly, why would we have such different times for warmup requests compared to cold starts? Shouldn't they be nearly identical?! Rock on, -Hardwick [Link 1] -http://stackoverflow.com/questions/9422698/ah-warmup-producing-hardde... On Jul 12, 12:26 pm, David Hardwick david.hardw...@bettercloud.com wrote: Hello, I realize there's been a lot of discussion on startup times exceeded on this forum recently, but wanted needed to post this experience we had this morning to keep the attention on this important issue. We uploaded a point release of our app to a not-live version this morning and, of course, we were going to click around on that instance to make sure it's all kosher before making that version live. The warm-up requests for the not-live version were exceeding the deadline limit of 60s... __and__we__are__on__F4s__!_!. However, the LIVE version of the app crashed too, 500 server errors, instance counts went to zero, all sorts of whacky stuff was seen in the control panel. All that happened to our LIVE version without when all we did was upload another non-live version and hit it with a single request...did I mention we were on F4s? ;-) Does the failure of any instance to exceed the 60s limit take down all instances to include live one? We did a few things as quickly as possible since our live application was down, so clearly we didn't have the time to take the scientific approach of only changing one thing at a time and wait to see if it that did it. We... 1. Switched from F4s to F2 (i figured if this would least get us on some new servers/instances) 2. Increased max idle instances from 1 to 2 (with F4s running, I'm fine with having just 1 idle instance and not at all happy about paying for 2 idle instances, so maybe we'll just increase this prior to deployments and then back down again after the deployment succeeds until we know more) 3. Made the recently uploaded version live (hey, why not, the production app was down for 10 minutes, so how much more harm could we do?) We use GWT and Guice, we jar everything (as I have been paying attention to this startup time discussions for quite some time now. We are also considering switching our Guice libraries to a non-AOP version as we saw suggested in another blog since we just need the injection. Any insight, and I'm all ears! app_id=s~myflashpanel Regards, -Hardwick -- *We make Google Apps even better.* *David Hardwick* *CTO* david.hardw...@bettercloud.com *Signature by Flashpanel http://flashpanel.com/* *See us in Mashable: Growing Up Google: How Cloud Computing Is Changing a Generation http://mashable.com/2012/04/30/generation-growing-up-google/* -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Java instance startup time out of control
My Project: # of classes in WEB-INF/classes: 1074 (jarred into a single jar for faster deployment) Size of WEB-INF/classes: 9.0M (jars down to 3.3M) # of jars in WEB-INF/lib: 87 Size of WEB-INF/lib: 90M Frameworks initialized: JDO PMF (3-6secs) JAXB single context (4-10 secs) Guice (2-4 secs) - just dipping my toes back in the water with guice on gae, only one experimental @Inject. Abandoned Guice previously due to initialization time concerns. JDO classes ~ 14 entity types JAXB beans ~ 146 Guice - single @Inject Fastest observed startup time: 11s Typical startup time: 20s Slowest startup time: 25s+ A few weeks ago I must have been moved to a new data center. Prior to that I was seeing between 2-3 times worse performance, for months. Both for instance initialization (which was 20-50+ secs), and general request latencies. This application is relatively small compared to some of the J2EE applications I've worked on in the past. /Tom On Jun 18, 1:34 pm, Jeff Schnitzer j...@infohazard.org wrote: This is incorrect. Guice does not perform classpath scanning, and while classpath scanning is nice for making JAX-RS @Path annotations work, it is optional and I have disabled it. The way it works is that you explicitly register the classes that have annotations. So each of them individually must be classloaded at startup. Despite the lack of classpath scanning, this process is still taking an excessive amount of time. Using Objectify is similar; all entity classes must be explicitly registered and introspected before datastore operations begin. So... this problem goes way beyond classpath scanning. The problem is getting classes loaded up front, and this problem doesn't expose itself until your application reaches a significant level of complexity. By this standard, Objectify is a heavyweight framework - the only way around loading entity classes up-front is to use the low-level API. I am impressed by your 3.5 second startup time. Does that include a datastore hit (ie Objectify registration)? How big is your project? Would you complete this straw poll, filling in your answers? Everyone else reading with Java instances, would you do the same? My project: # of classes in WEB-INF/classes: 619 (cd war/WEB-INF/classes; ls -R | grep class | wc -l) Size of WEB-INF/classes: 3.3M (cd war/WEB-INF/classes; du -sh .) # of jars in WEB-INF/lib: 54 Size of WEB-INF/lib: 42M (25M of this is GAE SDK) # of classes registered with Objectify: 36 (plus maybe half that again in @Embed and @Serialize classes) # of classes registered with other means (any explicit classloading, ie JAX-RS): 100+ Fastest observed startup time: 20s Typical startup time: 50s Slowest startup time: deadlined 60s+ I readily acknowledge that I have a fairly large number of jar dependencies. However, I'm not scanning them. They're also (almost) all essential for certain features; I do a lot of integration with third-party APIs. At best I can get rid of one or two by rewriting a few sections of code. Also... this project isn't really that big as enterprise projects go. I've worked with much, much larger codebases in the past. I shudder to think what that would do to appengine :-( Jeff On Mon, Jun 18, 2012 at 7:26 AM, Michael Hermus michael.her...@gmail.com wrote: Jeff, If by going back to Java programming circa 2002, you mean not using annotation processing that requires full classpath scanning, I think that is in fact the only solution right now. Based on my limited research, I think you really need to stay away from that in order to avoid cold start time problems with GAE Java. In fact, the 'Best Practices' section of the Objectify wiki is one of the first places I saw this. Although you say have no classpath scanning, the JAX-RS @Path processing is exactly that. In addition, I am almost sure that Guice is scanning the entire classpath for @Inject annotations as well. I wholeheartedly agree it stinks that you have to avoid leveraging awesome frameworks like those (and hence having better, more maintainable code) in order to have a properly functioning GAE Java application. I would gladly star any feature request you make in this direction. However, for now I am staying away. As a reference, I use only Objectify and a few other standard java libraries, but no heavyweight frameworks, and my _ah_warmup requests have been recently averaging about 3.5 seconds. -Mike On Saturday, June 16, 2012 5:56:33 PM UTC-4, Jeff Schnitzer wrote: We're having a big problem with instance startup time. It varies between 20s and 60+ seconds, and lately it's tending towards the high end. We're starting to experience downtime because instances get deadlined before they go active. This app is well optimized for GAE. There's no classpath scanning and it doesn't try to eagerly load data. On a good day it starts in 20s... so at this
[google-appengine] Re: Improved performance?
We're seeing something like a 3x reduction in average request latency. Pretty dramatic improvement. I'm pretty sure I used to see these numbers for our app as long as a year ago. The avg latency seemed to creep up over time until this reset. Instances are spinning up much more quickly too. On Jun 6, 12:32 pm, thecheatah thechea...@gmail.com wrote: Has anyone noticed significantly improver performance since the beginning of this month? Where a single instance handled about 4-5 requests per second, a single instance is now handeling 11-12! This has obviously lowered my billing :D From the beginning I expected a small single core instance to be able to handle about 10 requests/second. I always thought the over head of the datastore was very high. I hope Google continues to make these kinds of improvements. I will definitely consider using the Google App Engine for my future apps! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[appengine-java] Re: .A research: how long will your java app instance start up fully?
~9s (1/3 JDO PMF, 1/3 JAXB context, 1/3 other) On Sep 29, 9:29 pm, Tapir tapir@gmail.com wrote: ? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[google-appengine] Re: Are there any plans to increase the 150MB app size limit?
Are you deploying a .junit_symbolMaps folder (in your compiled GWT root)? If so, it's large, and can be safely deleted/excluded from the war before deploy. /Tom On Sep 14, 7:47 am, Daniel Florey daniel.flo...@gmail.com wrote: When using GWT the app can quickly exceed the 150MB app size limit. Are there any plans to increase this limit to let's say... unlimited? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Another HR Refactoring Issue - When to clear a cache?
What I try to do in these cases is a combination of: - when a new x is added, say Your x has been added successfully. It may take a few minutes to appear in the x list - In the x listing page, only show the most basic info for each (Ideally a name that doesn't change, but see next bullet). So a vanilla query accross entity groups is fine here. Some users may see the new x sooner than others (no biggie, matter of seconds usually) - If the name _can_ get changed, say something to the changer like The new name for your x may take a few minutes to appear in the x list - Clicking on the details of a x does a get-by-key, ensuring the latest And If you're going to cache the list page, have the cache maintained by a cron job, to ensure there are no collisions. Cron job needs to be a bit smart, since in theory it could also see a new x one run, then not see it on a query a minute later. But at least there is a single place to deal with that with no contention on the cache. /Tom On Sep 5, 1:58 pm, Joshua Smith joshuaesm...@charter.net wrote: Yeah, that'll work, too. Unless there are two changes to the list of boards at about the same time. Then it will give a bizarre result. The general idea of augmenting queries with extra results that you suspect it might not get seems to come up a lot as a solution to the eventual consistency problem. Other than entity groups (which doesn't seem to fit any of my real-world data models), this is the only suggestion in the docs. However, it really kinda sucks. Since query results are usually sorted in some way, you need to re-sort to get the fixed record into the right place. But, of course, you can't do that if you are using the query as an iterator. And, of course, the change may have made that record no longer fit the query criteria, or it may have caused it to fit where it didn't before, so you need to deal with all three cases: remove the record from the list of results, add the record to the list of results, and replace the record in a list of results. And then there is the question of where you put the record. Putting it into a cookie or query parameter is weird, and fragile. Putting it into memcache is fragile. I suppose it's possible that there just is no good design pattern to use here. In particular, I'm having trouble coming up with a way that doesn't require me to wrap 3 old lines of code with 20 lines of crap, then repeating that in 50 different places. I don't mind having a bunch of crap if there is some way to reuse it. But I'm struggling with that. Is ANYBODY out there happy with the way they solved this problem? -Joshua On Sep 4, 2011, at 5:51 PM, Tom Phillips wrote: I clear the cache whenever the list of boards changes in some way How about update the cache at that point instead of clearing it? Need be you could even generate the HTML for the cache update with a URLFetch to your UI handler where you include the added/changed board key(s) as parameters, so they can be gotten with strong consistency on that request and merge into the query result. /Tom On Sep 4, 11:16 am, Joshua Smith joshuaesm...@charter.net wrote: My monkeypatching solution (see my recent post in the -python group), which Guido says I shouldn't use, but which is just so darned pretty I can't help it, has gotten me through the first challenge of switching to HR, which is dealing with google search results containing keys into my old app's data store. So now I'm looking at the big Kahuna problem of consistency. Here's my first messy challenge there: My app puts a list of boards on the home page for a town, along with the list of meetings. Generating that list of boards was taking a lot of CPU, but they hardly ever change, so I put in a memcache system that built the HTML when it wasn't in the cache, and then cached it before serving. I clear the cache whenever the list of boards changes in some way. Well that ain't gonna work in HR. It's quite possible that I update a board, clear the cache, and someone comes and hits that page before eventually consistent comes to pass. So now I've got a cached copy of the stale data. (Note that I cannot use entity groups to solve this because some boards are municipal agencies, and therefore cannot be parented to the town that is building its list. I could parent all boards to some global parent, but, well, yuck.) I have some different ideas about how to fix this, but I'm wondering if anyone else who's done the port to HR has come up with a solution they find particularly elegant? I assume this is a pretty common problem, so there must be a design pattern out there… somewhere. Here are my ideas: - Clear the cache with a periodic task that re-clears it several times. I'm thinking a recurring geometric retry would be prudent (1, 2, 4, 8, 16, 32, 64, 128
[google-appengine] Re: Another HR Refactoring Issue - When to clear a cache?
I clear the cache whenever the list of boards changes in some way How about update the cache at that point instead of clearing it? Need be you could even generate the HTML for the cache update with a URLFetch to your UI handler where you include the added/changed board key(s) as parameters, so they can be gotten with strong consistency on that request and merge into the query result. /Tom On Sep 4, 11:16 am, Joshua Smith joshuaesm...@charter.net wrote: My monkeypatching solution (see my recent post in the -python group), which Guido says I shouldn't use, but which is just so darned pretty I can't help it, has gotten me through the first challenge of switching to HR, which is dealing with google search results containing keys into my old app's data store. So now I'm looking at the big Kahuna problem of consistency. Here's my first messy challenge there: My app puts a list of boards on the home page for a town, along with the list of meetings. Generating that list of boards was taking a lot of CPU, but they hardly ever change, so I put in a memcache system that built the HTML when it wasn't in the cache, and then cached it before serving. I clear the cache whenever the list of boards changes in some way. Well that ain't gonna work in HR. It's quite possible that I update a board, clear the cache, and someone comes and hits that page before eventually consistent comes to pass. So now I've got a cached copy of the stale data. (Note that I cannot use entity groups to solve this because some boards are municipal agencies, and therefore cannot be parented to the town that is building its list. I could parent all boards to some global parent, but, well, yuck.) I have some different ideas about how to fix this, but I'm wondering if anyone else who's done the port to HR has come up with a solution they find particularly elegant? I assume this is a pretty common problem, so there must be a design pattern out there… somewhere. Here are my ideas: - Clear the cache with a periodic task that re-clears it several times. I'm thinking a recurring geometric retry would be prudent (1, 2, 4, 8, 16, 32, 64, 128, 256, 512 seconds, and then pray that we have consistency) - Checksum the modified or new board, and put that sum into memcache. When generating the new board, confirm that any checksums are good. This seems more deterministic, except I don't trust memache not to squelch the checksum record. So perhaps I should do something in the datastore. This feels like it's be about 10x as much code as the stupid geometric flush. Any suggestions? -Joshua -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The scheduler needs a fix and quick.
Are you using JDO by any chance de Witte? There is a bad connection leak in DataNucleus (OMFContext) that results in major leaked memory. If you have non-transactional reads enabled in jdoconfig.xml (the default) you have this leak. Event when the request limit was only 15000, my instances grew their whole life up (on every read) to about ~130MB by the 15000 requests (and would have kept growing) After changing to: property name=javax.jdo.option.NontransactionalRead value=false / and wrapping all reads (including queries) in transactions, my instances hang steady at ~80-90MB. I'll keep it his way until the problem is fixed. I found the leak when profiling with JProfiler, and then noticed there was an old issue raised by the datastore guys on it that was since closed pending an eventual upgrade to Datanucleus 3.0. Can't seem to find the issue right now though. /Tom On Sep 2, 5:16 am, de Witte jcreator.xi...@gmail.com wrote: The app itself doesn't need much memory. The memory footprint is high because of the many active threads to handle the requests. Less memory means lesser threads means lesser requests to be handled asynchronously by a single instance. So hopefully they won't reduce it, where did you read it? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: The scheduler needs a fix and quick.
The memory issue is indeed tangental to the scheduler issues. A few of us just happened to notice it from de Witte's instance snapshot and figured we'd point it out as a separate issue and potential problem for him once the 128MB limit is enforced. I believe that that instance would already be killed off prematurely in the backend with an OutOfMemoryError. And will be in the frontend as well at some point (soon?). /Tom On Sep 2, 5:57 am, Bay troels...@gmail.com wrote: This has nothing to do with memory leaks. As I told before I experience the exact same problem and 1) the new limits are not in force yet 2) there are no mentions in the log that a soft memory limit has been reached (which there are, if you have a leak and it closes down instances as a consequence) 3) settings about max. idle instances and latency before new instance is spawned are already set to a configuration that should produce the least amount of instances I ask you please try not to derail the discussion from the fact that the scheduler is broken and needs 1) a quick fix in the form of a user-setting to allow only 1 instance being run (as the needless and passive instances spawned are costing a lot of money under the new regime for small apps) 2) a long-term fix so that it does not produce new instances that go around doing nothing (this should help larger apps as well) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Serious problem: Rollback of data on HRD
X and Y used only get-by-key, and read policy was unset in jdoconfig.xml, so using the default strong. These transactions were idempotent. BUT..I scoured the code again and sure enough, there is third entry point I missed in my HRD prep that reads the entity state via..you guessed it..a query accross entity groups. So there is a third transaction Z that can collide with Y, and it's the one writing stale state it sees from the query (state after X is what it overwrote with). I thought I had covered all of my HR migration tweaks - but missed this one entry point at least. Easily fixed now. Thanks, (and thanks for zigzag merge join BTW, it rocks) Tom On Aug 19, 9:34 pm, Alfred Fuller arfuller+appeng...@google.com wrote: Are your transactions idempotent? It is possible that the transaction is being run (and succeeding) twice in this case. What other request is colliding with first? You are not using any non-ancestor queries or setting read_policy=EVENTUAL on any reads correct? On Fri, Aug 19, 2011 at 12:42 PM, Tom Phillips tphill0...@gmail.com wrote: I'm seeing the same since moving to HR last week. It happens rarely, and the only clue is a ConcurrentModification in the logs (java in my case). Pure speculation, but it looks to me like some sort of background transaction retry might overwrite the entity with stale data, rather than a rollback. Scenario for me is like: pre) Entity bob has property height=70 1) thread 1, transaction X, height=75-commit() [appears to succeed] 2) Meanwhile (within a second or so) thread 2, transaction Y height=80- commit() [ConcurrentModificationException] - I pause 500ms and retry - commit() [appears to succeed this time] 3) For a while (up to a minute or so, but possibly much longer) all get-by-key on bob show height==80 (ok) 4) Another while later all get-by-key on bob suddenly show height==75, as per transaction X (not good!) My speculation is that the ConcurrentModification could sometimes indicate there was disruption of BOTH transaction X and Y, even though reported for Y. Perhaps X had gotten past commit() call but hadn't yet reached milestone A of http://code.google.com/appengine/articles/transaction_isolation.html, and was also (temporarily) aborted due to the contention. Then some sort of background retry on X sometimes (rarely) re-inserts it into the transaction queue BEHIND my explicit retry on Y, and eventually overwrites with the whole entity state from X in 1) And it appears that sometimes the background retry of X may not even happen till a good while later. Any chance something like this is happening? /Tom On Aug 16, 10:11 pm, Greg g.fawc...@gmail.com wrote: Please check your logs for a warning Transaction collision. Retying Something very similar is happening on my app, where DB put()s silently fail (equivalent to the entity being rolled back) very occasionally. This has only started happening after moving to HR. In my app, I get this warning very consistently (every time) at exactly the time the entity is supposed to be stored. I would be very interested to hear if you find this warning too. If so, I think it points to a bug in the transaction collision handler in put(). Please let me know! See my earlier post here: http://groups.google.com/group/google-appengine/browse_thread/thread/... Cheers Greg. On Aug 14, 10:21 pm, Raymond C. windz...@gmail.com wrote: I have recently ran into a problem after migrating to HRD: My application is a social online game which I have recently migrated from M/S to HR Datastore around 3 weeks ago. Since 2 weeks ago I have started receiving reports from players which their game progress are rolled back suddenly while playing, which progress made in the recent few days are missing. I have verified the problem through data on other entities (in different entity group) that the reports are actually legit and at least several days of progress are actually rolled back (with updates to the entities in the last few days are all missing). Player's data in the game are retrieved through id ( Player.get_by_id(player_id) ) and because the gap is so large (days) I believe it is not a problem on my code (nowhere in my code cache player's data). It has never happened before for nearly 1 year so I am guessing if it is related to HRD. I remember there was a thread here before which reported data being rolled back on HRD but I can not find it anymore. As you know with AppEngine datastore's distributed nature, it is so hard to monitor this kind of problem to ensure the problem exist. I would like to ask if anyone has ran into this problem as well or suspect that you have had this problem before with your HRD application? -- You received this message because you are subscribed to the Google Groups Google
[google-appengine] Re: Serious problem: Rollback of data on HRD
I'm seeing the same since moving to HR last week. It happens rarely, and the only clue is a ConcurrentModification in the logs (java in my case). Pure speculation, but it looks to me like some sort of background transaction retry might overwrite the entity with stale data, rather than a rollback. Scenario for me is like: pre) Entity bob has property height=70 1) thread 1, transaction X, height=75-commit() [appears to succeed] 2) Meanwhile (within a second or so) thread 2, transaction Y height=80- commit() [ConcurrentModificationException] - I pause 500ms and retry - commit() [appears to succeed this time] 3) For a while (up to a minute or so, but possibly much longer) all get-by-key on bob show height==80 (ok) 4) Another while later all get-by-key on bob suddenly show height==75, as per transaction X (not good!) My speculation is that the ConcurrentModification could sometimes indicate there was disruption of BOTH transaction X and Y, even though reported for Y. Perhaps X had gotten past commit() call but hadn't yet reached milestone A of http://code.google.com/appengine/articles/transaction_isolation.html, and was also (temporarily) aborted due to the contention. Then some sort of background retry on X sometimes (rarely) re-inserts it into the transaction queue BEHIND my explicit retry on Y, and eventually overwrites with the whole entity state from X in 1) And it appears that sometimes the background retry of X may not even happen till a good while later. Any chance something like this is happening? /Tom On Aug 16, 10:11 pm, Greg g.fawc...@gmail.com wrote: Please check your logs for a warning Transaction collision. Retying Something very similar is happening on my app, where DB put()s silently fail (equivalent to the entity being rolled back) very occasionally. This has only started happening after moving to HR. In my app, I get this warning very consistently (every time) at exactly the time the entity is supposed to be stored. I would be very interested to hear if you find this warning too. If so, I think it points to a bug in the transaction collision handler in put(). Please let me know! See my earlier post here:http://groups.google.com/group/google-appengine/browse_thread/thread/... Cheers Greg. On Aug 14, 10:21 pm, Raymond C. windz...@gmail.com wrote: I have recently ran into a problem after migrating to HRD: My application is a social online game which I have recently migrated from M/S to HR Datastore around 3 weeks ago. Since 2 weeks ago I have started receiving reports from players which their game progress are rolled back suddenly while playing, which progress made in the recent few days are missing. I have verified the problem through data on other entities (in different entity group) that the reports are actually legit and at least several days of progress are actually rolled back (with updates to the entities in the last few days are all missing). Player's data in the game are retrieved through id ( Player.get_by_id(player_id) ) and because the gap is so large (days) I believe it is not a problem on my code (nowhere in my code cache player's data). It has never happened before for nearly 1 year so I am guessing if it is related to HRD. I remember there was a thread here before which reported data being rolled back on HRD but I can not find it anymore. As you know with AppEngine datastore's distributed nature, it is so hard to monitor this kind of problem to ensure the problem exist. I would like to ask if anyone has ran into this problem as well or suspect that you have had this problem before with your HRD application? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[appengine-java] Re: Datastore access deadlock
To be defensive, you may want to try setting ~8 second timeouts for queries (setTimeoutMillis()) and for general read/writes in, for jdo, jdoconfig.xml (javax.jdo.option.DatastoreReadTimeoutMillis, javax.jdo.option.DatastoreWriteTimeoutMillis). You can then retry a couple times within the request keeping under the 30 second limit. It often, but not always, succeeds on one of the retries and salvages the request. /Tom On Jul 28, 5:11 pm, Eduardo Garcia Lopez egar...@stream18.com wrote: Hi, since last week I am experiencing a very serious problem in my application. Probably since I deployed for the first time after the new SDK 1.5.2 was released. The application is locking during 30 seconds (then it raises an timeout Exception) just doing a single query. The query starts running normally during a couple hours, then randomly starts locking, and after that almost all queries like this lock. This is the query: PersistenceManager pm = PMF.get().getPersistenceManager(); cmd = pm.getObjectById(Command.class, id); I have also tried changing to this query: PersistenceManager pm = PMF.get().getPersistenceManager(); query = pm.newQuery(Command.class, id == idParameter); query.declareParameters(Long idParameter); query.setRange(0,1); query.setUnique(true); cmd = (Command) query.execute(id); in both cases, it hangs in last sentence. The persistence class is NOT marked as detachable. The received exception is this: com.google.apphosting.api.ApiProxy$ApiDeadlineExceededException: The API call datastore_v3.RunQuery() took too long to respond and was cancelled Can somebody help me with this? Is somebody experiencing this error? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
When are threaded instances considered busy? If it is while they are serving only a one request it would explain why Always on instances for threaded Java are now severely under- utilized. /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained above. Thanks, galoch On Jul 21, 5:56 pm, Johan Euphrosine pro...@google.com wrote: After speaking with Engs, I think I can explain what is going on: Here are the current scheduling rules: ( reads as has priority for handling the incoming request) 1/ Idle Always On instance Spawning a new Dynamic instance 2/ Spawning a new Dynamic instance Busy Always On instance 3/ Idle Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance I will give you an example to illustrate the behavior you all noticed, that is Dynamic instance handling request
[google-appengine] Re: GAE starting unnecessary instances
The current behavior makes me suspect we are being prepared for Always- on being replaced completely by the new scheduler knobs. Being able to turn up the number of idle instances does make always-on somewhat redundant, as long as the idle instances stick around for a while. Also, if always-on instances WERE being properly utilized right now, it would require artificial load to ascertain the effects of the new scheduler on dynamic instances (assuming the three AO instances were sufficient for your app previously). Many devs wouldn't learn about the new scheduler attributes until after their app (suddenly) becomes popular - not the best time for surprises. And Google wouldn't get as much feedback on the features/behavior of the new scheduler. Is Always-on going to be kept under the new model? /Tom On Jul 22, 10:57 am, Johan Euphrosine pro...@google.com wrote: HI Galoch, Thanks for the followup, I think you are experiencing a combinaison fo the two following rules I was pointing to in my previous email: ( reads as has priority for handling the incoming request) 2/ Spawning a new Dynamic instance Busy Always On instance 4/ Idle Dynamic instance Idle Always On instance Applied to your example it could means that: Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr - 1 Hours ago while all your Always On instance were busy and you had a burst of incoming requests and the scheduler spawned new Dynamic instances as per rule 2/ highlighted above. - After the burst and back to normal traffic the new Dynamic Instances were handing incoming requests in priority as per rule 4/ highlighted above. - 2 Minutes ago all your instances Always On + Dynamic were busy again and the scheduler spawned a new Dynamic instance that handle 7 incoming requests. Hope that make more sense for you and Francois, but as I said earlier we are open to suggestion and I will make sure someone working on the scheduler team monitor this thread for your input. On Fri, Jul 22, 2011 at 9:09 AM, Galoch galoch...@gmail.com wrote: @Johan, The issue is not about Always On instance being busy. Its actually the other way ... the Always On instance is never busy ... at least that is what we observed in last 3-4 days. Your explanation may be partly true since this behavior keeps on changing. For e.g. I have a snapshot of instances from July 19th and here's the details (for some reason I can't see a link to attach the snapshot images here): Resident Instance 1: Requests: 49 Age: 1Hr Resident Instance 2: Requests: 6 Age: 1Hr Resident Instance 3: Requests: 2 Age: 1Hr Dynamic Instance 1: Requests: 7 Age: 2min Dynamic Instance 2: Requests: 291 Age: 1Hr Dynamic Instance 3: Requests: 322 Age: 1Hr This is under no load with only very light weight cron jobs running. This gets much much worse during the day under peak load with requests for dynamic instances reaching 1000+ in matter of minutes and resident instances have only 1 request served. As you see above Resident Instance 2 and 3 are hardly hit so I don't think they are busy at all. On the other hand, Dynamic Instance 2 and 3 get most of the hits. Dynamic Instance 1 is what is killing us. It keeps getting killed and reborn within that 5 minute window!! We use Spring framework and it is really very expensive for us when a new instance starts up. Just to give you a background, we had gone through a real roller coaster ride to make this to work on GAE by breaking the loading of framework into many different chunks. But still spinning was out of control. Then we found java threads to our rescue. We worked through the hack to load JDO to avoid UnsupportedOperationException. We finally got it to work where most of our requests were served by Always On instances with occasional spinning of Dynamic instances. It was quite impressive. Unfortunately, this was short lived when we hit this new behavior with GAE. The very last thing we want GAE to do is create a new instance every few minutes as it could easily reach 30 second deadline during the day and throw critical error. I am not sure when the new billing will come into effect but we really need this thing fixed as it literally brings down our app to a grinding halt. So I am open to any suggestions you guys think can help us. Another thought about new scheduler is to have a configurable schedule. For e.g. our users are mostly business users who work during normal business hours. We want to be able to spin more Always On instances during those hours and bring the number down during nights and weekends. Dynamic instances won't work for us due to reason explained
[google-appengine] Re: GAE starting unnecessary instances
Yes, seeing the same problem on all my apps. Java, M/S. Raise a production issue Francois? /Tom On Jul 19, 6:34 am, Francois Masurel f.masu...@gmail.com wrote: Even 3 now, for just just a few new requests. Geez. It's getting really expensive :-( Instances-VinoCities_3.jpg 43KViewDownload -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: GAE starting unnecessary instances
http://code.google.com/status/appengine shows they are investigating issue(s) with Java currently.. On Jul 19, 7:28 am, Tom Phillips tphill0...@gmail.com wrote: Yes, seeing the same problem on all my apps. Java, M/S. Raise a production issue Francois? /Tom On Jul 19, 6:34 am, Francois Masurel f.masu...@gmail.com wrote: Even 3 now, for just just a few new requests. Geez. It's getting really expensive :-( Instances-VinoCities_3.jpg 43KViewDownload -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Do I really need to switch to HR?
Hi Robert, My example is the standard app engine encoded key. The impression I got from Ikai was that that was fine - it was any custom encoding or hashing (sha1 for example) that was an issue. Maybe Ikai could confirm whether the unmodified encoded key will be converted from my example? If not, I'm a long way from being ready for HR. /T On Jul 11, 1:23 am, Robert Kluin robert.kl...@gmail.com wrote: On Fri, Jul 8, 2011 at 15:30, Tom Phillips tphill0...@gmail.com wrote: That aliasing via tools sounds pretty good to me. I was worried about the timing of aliasing my old app to my new one if it had to be done via raising a billing issue. Having the ability to time it ourselves is much better. So if I understand the key conversion correctly, it sounds pretty good too. It will navigate every possible value in my data looking for and converting [encoded] keys. But just making sure - it would detect and convert both of these JDO properties: Your data gets copied to a new app; the conversion happens during that copy operation. @Persistent private SetKey favoriteFoods; @Persistent private SetString favoriteFoodsEnc; //Encoded key strings, such as ag1jbGluaWNvbmV4YXBwcg8LEghQcmFjdGljZRjpBww Unless I'm mistaken, they've said this case is NOT converted. Correct me if I'm wrong about that example. Externalized keys being used by clients is a bit of an issue for me too. I'll need to deal those old keys at the edges of the system I guess. I wish I had noticed earlier on that the app id is encoded into keys - that would have raised red flags about exposing them to clients. Actually, I should have been more suspicious about exposing a database key directly to a client no matter what its format, oops. In some cases exposing the key is the easiest solution, like if you're using entity groups. But I'm sure a lot of people will be caught by this issue; I know I use encoded keys in external systems to cross-reference data to App Engine apps. Luckily, while looking into using namespaces, I happened to have written a simple encoder that does not included the appid (and only optionally the namespace) and converted my apps to this format. /Tom On Jul 8, 3:17 pm, Ikai Lan (Google) ika...@google.com wrote: The tool *should* preserve keys that are stored as straight keys. That is - Keys stored in list, String or reference properties. The tool should read these, decode them, change the app ID (so you don't get the exception about reading data from a different app ID), and resave these to the new datastore. What the tool does NOT do is smart decoding of keys. Some people do things like sha1 the keys; this functionality will 100% break since the keys will be different. Remember that keys encode the application ID in them. If the application ID changes, the hashes will change. The tools will not allow migration within the same app ID per se. You will need to create a new app ID. The tools will allow you to map an alias from the old app ID to the new one, so for any routing intents and purposes, the old app ID will still point to the new old. Internally, however, the new app ID will be stored. This matters if you depend on any functionality that reads the current app ID from system environment variables in production. All new HR apps, for instance, prefix a s~ before the app ID. Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Thu, Jul 7, 2011 at 11:55 PM, Tom Phillips tphill0...@gmail.com wrote: Greg, can you reveal in advance whether the new tools will allow for migration within the same app id and/or preserve encoded entity keys? /Tom On Jul 6, 8:01 pm, Gregory D'alesandre gr...@google.com wrote: We are working on better tools for migrating to HRD (and they are currently being tested), I'll post once we have them widely available. Greg On Wed, Jul 6, 2011 at 5:52 PM, Waleed Abdulla wal...@ninua.com wrote: Please star this issue if you agree that Google should make the migration process easier rather than putting the burdon on the developers. After all, most developers signed up to GAE to avoid having to deal with issues like that. http://code.google.com/p/googleappengine/issues/detail?id=5250 On Wed, Jul 6, 2011 at 8:39 AM, Robert Kluin robert.kl...@gmail.com wrote: Keys do contain the appid. One solution would be to adjust your code to catch the exception that gets thrown when accessing the old apps data, the recreate the key for the new appid. Robert On Wednesday, July 6, 2011, Joshua Smith joshuaesm...@charter.net wrote: Another thing that occurs to me is that anyplace I've used a key external to the application would have
[google-appengine] Re: Using application versioning with a custom domain
See this post http://groups.google.com/group/google-appengine/msg/2529ebea7090f8fe Works for backend addressing so I can't see why it wouldn't for versions. /Tom On Jul 9, 12:03 am, Jason jmitche...@gmail.com wrote: Hello App Enginers, One useful feature in App Engine is accessing different versions of the application via the domain name - ie 1.myapp.appspot.com, 2.myapp.appspot.com, etc. I'm posting to ask ways to do this with custom domains. Eg 1.myapp.com, 2.myapp.com, etc. The first way that pops to mind is at the domain name level, so 1.myapp.com is a cname for 1.myapp.appspot.com. Although that sounds like it would be a DNS headache, if it would work at all? A second thing that comes to mind is to servlet map *.myapp.com and inside the servlet use servlet fowarding to 1.myapp.appspot.com. Again I'm not really certain of the merits of this. Any thoughts or experiences would be appreciated! Just for some context, the basic thing I want is to release trunk to a different sub domain for testing on a small group of people before it gets pushed to the main user base. Thank you -Jason -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Do I really need to switch to HR?
Greg, can you reveal in advance whether the new tools will allow for migration within the same app id and/or preserve encoded entity keys? /Tom On Jul 6, 8:01 pm, Gregory D'alesandre gr...@google.com wrote: We are working on better tools for migrating to HRD (and they are currently being tested), I'll post once we have them widely available. Greg On Wed, Jul 6, 2011 at 5:52 PM, Waleed Abdulla wal...@ninua.com wrote: Please star this issue if you agree that Google should make the migration process easier rather than putting the burdon on the developers. After all, most developers signed up to GAE to avoid having to deal with issues like that. http://code.google.com/p/googleappengine/issues/detail?id=5250 On Wed, Jul 6, 2011 at 8:39 AM, Robert Kluin robert.kl...@gmail.comwrote: Keys do contain the appid. One solution would be to adjust your code to catch the exception that gets thrown when accessing the old apps data, the recreate the key for the new appid. Robert On Wednesday, July 6, 2011, Joshua Smith joshuaesm...@charter.net wrote: Another thing that occurs to me is that anyplace I've used a key external to the application would have to be dealt with. For example, one of our apps has an RSS feed that our managers use to keep track of whether customers have uploaded new information. This looks like: http://myapp.com/rss?cust=DAFHP983RPFDSALFHDSKLFJHLSDKAFH (where the gobledegook is the datastore key) My managers would need to update all their bookmarks, since I believe that key has the name of the app pickled in it someplace, right? Is there any way to avoid having the keys change when I switch to HR? If not, is there a way that I could decorate database queries so that they quietly turn old keys into new ones? -Joshua On Jul 5, 2011, at 3:35 PM, Joshua Smith wrote: I have several apps and I've got this pesky to-do list item for all of them, to switch them over to the HR datastore. In preparation, I've read blog entries by people who have done this, and I'm still on the fence about whether it is worth the trouble. I get very few datastore timeouts in my apps, and periodic maintenance has never really been an issue. I've read about gotchas with eventual consistency, and it seems unlikely that I could convince myself that I'm safe from this for any reasonably complex app. (I'm not concerned that I cannot reprogram the apps to fix the problem, but rather I'm concerned that I could not ensure there weren't going to be problems. If it ain't broke...) I'm not clear on how I would go about getting google to grandfather my old 2000 emails quotas if I create a new -hr version of my app. I've also read that migrating uses TONS of CPU, and it is therefore going to cost me money to do. Really, the only reason TO do the switch is that I've read repeated admonitions from google that I *should* be using HR. What's the consensus here? Is it worth the time, risk, and cost? -Joshua -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- -- Robert Kluin Ezox Systems, LLC -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Do I really need to switch to HR?
That aliasing via tools sounds pretty good to me. I was worried about the timing of aliasing my old app to my new one if it had to be done via raising a billing issue. Having the ability to time it ourselves is much better. So if I understand the key conversion correctly, it sounds pretty good too. It will navigate every possible value in my data looking for and converting [encoded] keys. But just making sure - it would detect and convert both of these JDO properties: @Persistent private SetKey favoriteFoods; @Persistent private SetString favoriteFoodsEnc; //Encoded key strings, such as ag1jbGluaWNvbmV4YXBwcg8LEghQcmFjdGljZRjpBww Correct me if I'm wrong about that example. Externalized keys being used by clients is a bit of an issue for me too. I'll need to deal those old keys at the edges of the system I guess. I wish I had noticed earlier on that the app id is encoded into keys - that would have raised red flags about exposing them to clients. Actually, I should have been more suspicious about exposing a database key directly to a client no matter what its format, oops. /Tom On Jul 8, 3:17 pm, Ikai Lan (Google) ika...@google.com wrote: The tool *should* preserve keys that are stored as straight keys. That is - Keys stored in list, String or reference properties. The tool should read these, decode them, change the app ID (so you don't get the exception about reading data from a different app ID), and resave these to the new datastore. What the tool does NOT do is smart decoding of keys. Some people do things like sha1 the keys; this functionality will 100% break since the keys will be different. Remember that keys encode the application ID in them. If the application ID changes, the hashes will change. The tools will not allow migration within the same app ID per se. You will need to create a new app ID. The tools will allow you to map an alias from the old app ID to the new one, so for any routing intents and purposes, the old app ID will still point to the new old. Internally, however, the new app ID will be stored. This matters if you depend on any functionality that reads the current app ID from system environment variables in production. All new HR apps, for instance, prefix a s~ before the app ID. Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Thu, Jul 7, 2011 at 11:55 PM, Tom Phillips tphill0...@gmail.com wrote: Greg, can you reveal in advance whether the new tools will allow for migration within the same app id and/or preserve encoded entity keys? /Tom On Jul 6, 8:01 pm, Gregory D'alesandre gr...@google.com wrote: We are working on better tools for migrating to HRD (and they are currently being tested), I'll post once we have them widely available. Greg On Wed, Jul 6, 2011 at 5:52 PM, Waleed Abdulla wal...@ninua.com wrote: Please star this issue if you agree that Google should make the migration process easier rather than putting the burdon on the developers. After all, most developers signed up to GAE to avoid having to deal with issues like that. http://code.google.com/p/googleappengine/issues/detail?id=5250 On Wed, Jul 6, 2011 at 8:39 AM, Robert Kluin robert.kl...@gmail.com wrote: Keys do contain the appid. One solution would be to adjust your code to catch the exception that gets thrown when accessing the old apps data, the recreate the key for the new appid. Robert On Wednesday, July 6, 2011, Joshua Smith joshuaesm...@charter.net wrote: Another thing that occurs to me is that anyplace I've used a key external to the application would have to be dealt with. For example, one of our apps has an RSS feed that our managers use to keep track of whether customers have uploaded new information. This looks like: http://myapp.com/rss?cust=DAFHP983RPFDSALFHDSKLFJHLSDKAFH (where the gobledegook is the datastore key) My managers would need to update all their bookmarks, since I believe that key has the name of the app pickled in it someplace, right? Is there any way to avoid having the keys change when I switch to HR? If not, is there a way that I could decorate database queries so that they quietly turn old keys into new ones? -Joshua On Jul 5, 2011, at 3:35 PM, Joshua Smith wrote: I have several apps and I've got this pesky to-do list item for all of them, to switch them over to the HR datastore. In preparation, I've read blog entries by people who have done this, and I'm still on the fence about whether it is worth the trouble. I get very few datastore timeouts in my apps, and periodic maintenance has never really been an issue. I've read about gotchas with eventual consistency, and it seems unlikely that I could convince myself
[google-appengine] Re: Updated App Engine Pricing FAQ!
Re: http://code.google.com/p/googleappengine/issues/detail?id=4834#c13 FYI I don't think that fix works for JDO via jdoconfig.xml. I've added a comment after reverifying it just now. I can easily reproduce it under load with or without the suggested settings. On Jun 30, 2:06 am, Ikai Lan (Google) ika...@google.com wrote: Oy, this is my fault for fanning the flames. I apologize for letting this topic catch fire in the wrong places. Before we go into more name calling: Vivek has been an App Engine developer for a while, almost from the beginning, while Jeff maintains one of the most popular Java libraries for GAE - Objectify - and trust me when I say that he *does* know what he is talking about. My statement called into question the assertion that PHP was what App Engine needed. While the long term, we-could-do-this-if-we-had-infinite-resources goal is certainly to support EVERY language conceivable to man on App Engine, this is probably something that Google will not do in the near term timeframe. More developers does not translate into a sustainable pricing strategy. Believe it or not, not everything gets cheaper just because you have more users. Some things actually get more expensive. Official support for PHP is one of these things. App Engine was an internal tool before it was an external one. We launched Python because we understand it. We launched Java because we understand it, and there is a huge userbase. Some Java users expected to be able to port their knowledge of Spring/JSF/Wicket/etc immediately over to App Engine without having to learn anything new. I think we set the expectations incorrectly here. The plan that is in place will be very close to what we launch with, because when we looked at different pricing plans, our analysis of previous usage trends and billing led us to believe that the one we have announced was the most balanced in terms of being developer friendly as well as sustainable. Unfortunately, we did understand that the changes would not work for some people. The most constructive discussion we can have right now is around how we can make this pricing work. What tools can we provide? What data do we not display? How should support work? And so forth. Throttler knobs, for instance, are an example of a feature where much of the requirements were sourced from constructive user feedback. Raising the priority of Python concurrency was another one. To answer the JDO question: have you tried comment #13? Seems to resolve the issue:http://code.google.com/p/googleappengine/issues/detail?id=4834#c13 Ikai Lan Developer Programs Engineer, Google App Engine Blog:http://googleappengine.blogspot.com Twitter:http://twitter.com/app_engine Reddit:http://www.reddit.com/r/appengine On Thu, Jun 30, 2011 at 3:11 AM, Branko Vukelic bg.bra...@gmail.com wrote: On Wed, Jun 29, 2011 at 8:29 PM, Jeff Schnitzer j...@infohazard.org wrote: On Wed, Jun 29, 2011 at 11:14 AM, vivpuri v...@vivekpuri.com wrote: @Jeff i dont you have the development experience on AppEngine to even take part on this discussion. Before suggesting, first go an check what Quercus does and can enable you to do on AppEngine. This is the stupidest thing anyone has said to me in years. And more to come if you keep replying to him. :) -- Branko Vukelić bra...@herdhound.com Lead Developer Herd Hound (tm) - Travel that doesn't bite www.herdhound.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[appengine-java] Re: Should JDO use be synchronized in threadsafe=true application?
I have only seen problems with getObjectById() (running threadsafe for
two months), as per
http://code.google.com/p/googleappengine/issues/detail?id=4834.
I tried the initialize with a synchronized dummy query but didn't
have much luck with it.
In the meantime I've created wrapper methods that let me switch
between using getObjectById(), and using a query by key. No other
changes - no synchronization of anything, including the
PersistenceManager. e.g.:
static private final boolean QUERY = true;
...
static public MyEntity getMyEntityById(PersistenceManager pm, Key
key) {
MyEntity e = null;
if (!QUERY) {
e = pm.getObjectById(MyEntity.class, key);
} else {
final Query query = pm.newQuery(MyEntity.class);
query.setFilter(key == targetKey);
query.declareParameters(String targetKey);
final String keyStr = KeyFactory.keyToString(key);
final ListMyEntity results = (ListMyEntity)
query.execute(keyStr);
if (results.size() != 0) {
e = results.get(0);
}
if (e == null) {
throw new AppException(FaultCodes.FAULT_OBJECT_NOT_FOUND,
Entity not found);
}
}
return e;
}
No issues with threadsafety for the query method and I can switch back
to getObjectById() easily to test if it's been fixed.
I haven't seen any issues so far (knock on wood) with writes or
deletes, even under high load - only with reads via getObjectById().
/Tom
On Jun 23, 6:12 pm, David turntwo...@gmail.com wrote:
Aaron (or others),
Do you have any updates from your experiences using JDO with
threadsafe enabled? Has your synchronization changes eliminated these
errors? Is it necessary to synchronize reads such as query.execute
and pm.detachCopyAll in additional to the write operations such as
pm.deletePersistent and pm.makePersistent?
Thank you,
David
On May 17, 4:51 pm, Aaron Shepherd ans...@onfast.com wrote:
Ok. My change to PMF.java to synchronize the get of the
PersistenceManager did NOT correct the problem.
Going to fallback to synchronizing every usage of the
PersistenceManager (reads and writes). sigh/
Will update this issue with any further results.
On May 17, 1:54 am, Stephen Johnson onepagewo...@gmail.com wrote:
Ah, I missed the point about it being tasks and didn't realize the
threadsafe only applied to user requests and not tasks. If indeed that
was
the case.
On Mon, May 16, 2011 at 10:44 PM, Juha K juha.kosk...@gmail.com wrote:
The datanucleus issue has a comment Then, when my tasks run in
parallel,
initialisation has already been done and the problem doesn't appear any
more., so before the threadsafe property, only tasks were run in
parallel.
Probably that's why I didn't see this error before, I didn't have tasks
doing db access.
--
You received this message because you are subscribed to the Google
Groups
Google App Engine for Java group.
To post to this group, send email to
google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Confused about loading vs. non-loading requests when threadsafe on
Hi Jon,
I raised issue http://code.google.com/p/googleappengine/issues/detail?id=5214
on the non-loading requests resulting in instance deaths with
threadsafe on. It was a bit more verbose than it probably needs to be,
but it describes the problem.
If anyone knows another way to detect loading requests that works with
threadsafe on, let me know. There is no guarantee that the servlet
context is loaded during the loading request, so that's out for now,
but if anyone knows another trick it will create options to work
around the problem until a fix is available.
Otherwise, I can disable threadsafe for now. But I'm reluctant - the
efficiency is strangely reassuring..
/Tom
On Jun 21, 1:25 pm, Jon McAlister jon...@google.com wrote:
Hi Tom,
I'm not sure about the servlet context listener code.
However, thanks for pointing out the behavior of the interaction of
loading requests and threadsafe apps. We are indeed sending down
non-loading-requests while the instance is still processing the
loading request. This is problematic (for the reasons you list) and
something we didn't anticipate. We'll try to get a fix in to one of
the next releases. If you file it as an external issue we can use that
to notify when its closed. Until then your suggested workaround is
good, the other known workaround being to turn off threadsafe.
Thanks for bringing this to our attention.
On Sun, Jun 19, 2011 at 5:11 PM, Tom Phillips tphill0...@gmail.com wrote:
The GAE documentation says you can detect loading requests like this:
// web.xml snippet
listener
listener-class
com.example.LogLoadingRequest
/listener-class
/listener
// LogLoadingRequest.java
public class LogLoadingRequest implements ServletContextListener {
private static final Logger logger =
Logger.getLogger(LogLoadingRequest.class.getName());
public void contextInitialized(ServletContextEvent sce) {
logger.log(Level.INFO, Loading request occuring.);
}
public void contextDestroyed(ServletContextEvent sce) {
}
}
But I'm finding from my logs that what GAE thinks is a loading request
(identified in log as loading_request=1) and which is given longer
deadline, does NOT always match the request that logs the above
Loading request occuring. when Threadsafe is on.
Should it, or is this method obsolete for threadsafe instances? Is
there some other way I can detect loading vs non-loading requests with
threadsafe on?
With threadsafe on, non-loading requests are being assigned by GAE to
instances while the loading request is still happening, resulting in
all sort of (Hard)DeadlineExceeded issues in the non-loading requests.
Reason is, loading requests are showing huge slow down and variance in
loading times lately, often exceeding the 30 seconds that non-loading
request are allowed - even though in my case loading should only take
about 4 seconds (jaxb,PMF) plus whatever classloading and jvm startup
takes. Worst thing is, these exceptions bring down the instance before
loading can even finish, requiring multiple startup attempts. I'm
trying to work around it by detecting loading vs. non-loading requests
and in non-loading either doing a no-op if not critical, or proxying
via urlfetch through to a backend until the loading request is
complete.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group
athttp://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[google-appengine] Re: Why a big difference between execution time and response time?
Hi Waleed, Because I've found tasks to be sometimes unreliable and unpredictable performance wise, I've implemented a strategy of engaging backend instance(s) as alternative non-task-based job processors. Basically, I now have two job processing engines that work differently and together offer redundancy and better performance. One (the traditional one) uses tasks, and the other doesn't - and I run them simultaneously. If one isn't working well for some reason, then at least the other is getting work done. In my case reliability is more of an issue than achieving very high throughput, but this could also work to increase overall throughput. It may not work quite as well for python as it does for java where I can have multiple worker threads running on each backend. And it isn't necessarily cheap. But it can pick up some slack for the task infrastructure, and if the cost allows you could always add more backend instances. With threadsafe in java land, it is even better since the backend is more fully utilized with multiple worker threads working on jobs simultaneously. Basically the implementation is to have one or more backend worker threads/instances that loop eternally looking look for jobs in the DB, grabbing batches of them and running them directly (with a short configurable pause if none are found in a cycle). Meanwhile, the other task-based processor is being driven by a cron job, which every minute essentially runs the same find me jobs code the backends use. Instead of running the jobs directly in the frontend, this one dispatches them out as tasks using DB cursors and task chaining to handle larger numbers of waiting tasks. It can create some contention on job records in the DB, though there should be ways you could minimize that if it becomes excessive. And as long as you mark newly found jobs as assigned within the same transaction that you query them out, you can avoid accidentally running the same job twice. Careful if you're on high-replication - you may need to account for consistency limitations while querying and assigning jobs (I'm on M/S). /Tom On Jun 21, 8:36 pm, Waleed Abdulla wal...@ninua.com wrote: Thank you, Nick. I did check your post before asking and pending_ms was missing. Also, I'm noticing throttle_code=2 in the log of some tasks! It would be great to cover that one as well. I've spent the last 3 days trying everything I can think of to make my app scale up, but GAE is working against me. I make the tasks longer, and GAE starts running them less often. I make them shorter, and my queues start to have a backlog. GAE simply doesn't want to scale up for me (started 3 days ago). I'm running out of ideas to try. Would appreciate your help. More details about what I tried here: http://groups.google.com/group/google-appengine/browse_thread/thread/... Regards, Waleed On Tue, Jun 21, 2011 at 5:04 PM, Nick Johnson (Google) nick.john...@google.com wrote: Hi Waleed, You may find this blog post useful for explaining the various fields of log entries: http://blog.notdot.net/2011/06/Demystifying-the-App-Engine-request-logs pending_ms is the amount of time the request spent waiting on an app server to become available to execute the task. -Nick Johnson On Wed, Jun 22, 2011 at 8:45 AM, Waleed Abdulla wal...@ninua.com wrote: Good catch, thanks. I'm guessing that pending_ms is the time a request stays on hold before execution? I couldn't find it documented anywhere. If so, then the question is: why would a request be in a pending state? Is it waiting for an instance to be free? This might also be related to my other thread about the task queue not scaling up [1]. Thanks, Waleed [1] http://groups.google.com/group/google-appengine/browse_thread/thread/... On Tue, Jun 21, 2011 at 3:33 PM, Nicholas Verne nve...@google.comwrote: One other note: In a large number of cases, if you the pending_ms (when it exists) to your measurement, you come close to the ms value you circled at the top. Nick Verne On Wed, Jun 22, 2011 at 8:18 AM, Nicholas Verne nve...@google.com wrote: Waleed, I'm going to have to ask one of the other infrastructure engineers about the log lines. The value you have circled (18386ms) is not always greater than the cpu_ms. I'm not sure what this value indicates. Nick Verne On Wed, Jun 22, 2011 at 6:27 AM, Waleed Abdulla wal...@ninua.com wrote: Nicholas, I implemented the change you suggested (making task_t0) a global variable, but that didn't change the calculations. I'm attaching another screen shot. There is a big difference between the actual processing time and the time reported in the logs. It doesn't happen in all tasks, but it's often enough that it's slowing down my application considerably and causing a lot of user complaints. I also calculated the total time to load the module (including time for
[appengine-java] Confused about loading vs. non-loading requests when threadsafe on
The GAE documentation says you can detect loading requests like this:
// web.xml snippet
listener
listener-class
com.example.LogLoadingRequest
/listener-class
/listener
// LogLoadingRequest.java
public class LogLoadingRequest implements ServletContextListener {
private static final Logger logger =
Logger.getLogger(LogLoadingRequest.class.getName());
public void contextInitialized(ServletContextEvent sce) {
logger.log(Level.INFO, Loading request occuring.);
}
public void contextDestroyed(ServletContextEvent sce) {
}
}
But I'm finding from my logs that what GAE thinks is a loading request
(identified in log as loading_request=1) and which is given longer
deadline, does NOT always match the request that logs the above
Loading request occuring. when Threadsafe is on.
Should it, or is this method obsolete for threadsafe instances? Is
there some other way I can detect loading vs non-loading requests with
threadsafe on?
With threadsafe on, non-loading requests are being assigned by GAE to
instances while the loading request is still happening, resulting in
all sort of (Hard)DeadlineExceeded issues in the non-loading requests.
Reason is, loading requests are showing huge slow down and variance in
loading times lately, often exceeding the 30 seconds that non-loading
request are allowed - even though in my case loading should only take
about 4 seconds (jaxb,PMF) plus whatever classloading and jvm startup
takes. Worst thing is, these exceptions bring down the instance before
loading can even finish, requiring multiple startup attempts. I'm
trying to work around it by detecting loading vs. non-loading requests
and in non-loading either doing a no-op if not critical, or proxying
via urlfetch through to a backend until the loading request is
complete.
--
You received this message because you are subscribed to the Google Groups
Google App Engine for Java group.
To post to this group, send email to google-appengine-java@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[google-appengine] Re: 500 Errors (DeadlineExceededError) on module imports and other random bits of code
I've been seeing more and more of these DeadlineExceededError exceptions over time especially in warmup requests. My /_ah/warmup should finish in about 6s (PersistenceManagerFactory, Jaxb). And I'm seeing these exceptions on user requests too even though the average latency is usually ~50ms. Java, Master/Slave. Not sure with these problems how my app will scale beyond the Always- on instances. On Jun 9, 2:31 pm, PandaSuit pandas...@gmail.com wrote: Anyone else seeing a large amount of 500s (DeadlineExceededError) lately on module imports? Over the last few days parts of my application that have never generated errors (and have not changed recently) have been throwing 500 errors and timing out. Often it is the first import of a module that times out or it may time out in the middle of an SDK function. It is usually never the same line of code from error to error either. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Task queue not executing
Thanks Mickey.
Was this perhaps also a not-so-gentle push for us heavyweight Java M/S
guys to move our stuff to HRD and backends? ;)
Seriously though, I think I'm going to add the option in my app to
startup backend instance(s) that do something like this:
while (tasksAreSlackingOff() haveMoreWork()
(!LifecycleManager.getInstance().isShuttingDown()) {
grabAJobToPickupSomeOfTheSlackFromThoseTaskSlackers();
saveState();
}
Better safe than sorry.
/Tom
On Jun 7, 4:32 am, Mickey Kataria (Google Employee)
mic...@google.com wrote:
Update: There was an issue with how the app servers were throttling
requests from task queues. It has only impacted Java apps running
with the master/slave datastore. We've put in a temporary fix that
alleviates the problem and will roll out a permanent fix in the next
day or so.
On Jun 7, 10:52 am, Mickey Kataria (Google Employee)
mic...@google.com wrote:
We're aware there is an issue and are looking into it. I'll post an
update as I have more info.
On Jun 7, 8:30 am, John Wheeler j...@highvolumeseller.com wrote:
This may or may not be related, but we're getting
ExclusiveLockError
UnknownError
and FinalizationError
when using Mapreduce which uses the TQ heavily. We never encountered these
errors before today which leads me to believe that the errors above are
related.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Task queue not executing
Yes, same problem here. This happened during the day a couple days last week too. Tasks sit waiting to be dispatched even though none are running and the frontend instances are lightly loaded with low latency. A task runs once in a while, sporadically. Cron jobs (once per minute in my case) also fail, only succeeding one in a while. They are on time Failed, and my handler isn't even invoked when they fail. I'm assuming cron is using the same task infrastructure. Raised production issue http://code.google.com/p/googleappengine/issues/detail?id=5150 on this. App engine does a great job scaling user-facing requests, but I'm starting to wonder if when App Engine as a system overall is under heavy load it is doing so at the expense of non-user-facing requests. My app is lightly loaded now, but that will change - do I need to be worried that if my app is heavily asynchronous, with throughput and timeliness depending on smooth task queue operation, I will run into big problems at scale on days like today? On Jun 6, 1:58 pm, tempy fay...@gmail.com wrote: Tasks for my app (ID: openkleio) are not executing, just sitting there and executing every couple of minutes, sometimes. I can run them manually just fine from the control panel. No quota issues, no errors - just sitting there. Anyone else seeing this? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Task queue not executing
It would be great if someone from Google could clarify if today's behavior is what we can expect out of task queues and cron. I noticed that depending on when you refresh the App Engine status page today http://code.google.com/status/appengine you do see Elevated for task queues (and at other times, Normal). If this flavor of Elevated may happen a few times a week, deal, I'll need to figure out a workaround or look at hosting at least the non- user-facing part of my app elsewhere. Maybe resident backend instances scanning the DB for jobs to run...ughh..seems like that would get expensive would be more cumbersome to tune than task queues. On Jun 6, 4:51 pm, tempy fay...@gmail.com wrote: Tom - I don't really see why heavy GAE-wide load should impact task queues or cron jobs in this way. The tasks run within our existing instances, and each instance, once it exists, has reserved a slice of resources for its own use. If that instance is lightly loaded (as mine are), it should be able to handle the task just fine, just as it would a regular request - isn't that the whole point of web hooks? The tasks are no different from ordinary requests, as far as the instances are concerned. I always assumed that the way system-wide load would manifest would be in the unavailability of new instances for scaling. Of course, these are educated guesses and I'm probably wrong. And I share your worries - my app is heavily asynchronous and large chunks of it come to a screeching halt when the tasks stop operating - I know task queues are advertised as time of execution is not guaranteed, but I hope that this does not include tasks should execute immediately, but may execute in ten minutes, or an hour, or tomorrow. If that's the case I'm really on the wrong platform. On Jun 6, 10:38 pm, Tom Phillips tphill0...@gmail.com wrote: Yes, same problem here. This happened during the day a couple days last week too. Tasks sit waiting to be dispatched even though none are running and the frontend instances are lightly loaded with low latency. A task runs once in a while, sporadically. Cron jobs (once per minute in my case) also fail, only succeeding one in a while. They are on time Failed, and my handler isn't even invoked when they fail. I'm assuming cron is using the same task infrastructure. Raised production issuehttp://code.google.com/p/googleappengine/issues/detail?id=5150 on this. App engine does a great job scaling user-facing requests, but I'm starting to wonder if when App Engine as a system overall is under heavy load it is doing so at the expense of non-user-facing requests. My app is lightly loaded now, but that will change - do I need to be worried that if my app is heavily asynchronous, with throughput and timeliness depending on smooth task queue operation, I will run into big problems at scale on days like today? On Jun 6, 1:58 pm, tempy fay...@gmail.com wrote: Tasks for my app (ID: openkleio) are not executing, just sitting there and executing every couple of minutes, sometimes. I can run them manually just fine from the control panel. No quota issues, no errors - just sitting there. Anyone else seeing this? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: pointing custom domain to google backend instance
If you just looking to de-appspot things and are OK with backendName.yoursub.yourdomain.com, this worked for me. Under your Google app settings for that application, add an entry with a wildcard to Your users can access [your app] at like this: http://*.yoursub.yourdomain.com where you presumably have http://yoursub.yourdomain.com for your frontend already (keep that one too). Configure DNS by adding a MX record at your domain provider for each backend (or wildcards if they support it - GoDaddy doesn't). E.g., pointing backendName.yoursub.yourdomain.com to ghs.google.com. Wait for the DNS change to propagate, then it should work. /Tom On Jun 2, 7:59 pm, radzish radz...@gmail.com wrote: Hello, I would like to expose one of my backend so it accepts client requests. I configured it adding public option. Everything works fine via backendName.appName.appspot.com. Now I wont to point my custom domain to that backend. I tried to do so the way it described in documentation of adding custom domains, but with no luck. new created domain points to default version of my apprather than to backend instance. How can it be resolved? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Something up with cron and task services last couple days?
Since at least yesterday, all of my cron jobs have been failing with on time - Failed up to half the time they are [supposedly] run. This is on all my apps, some of which have been updated to 1.5 and others still running 1.4.X. From my logs, it looks like the handler servlet (I'm using java) isn't even being entered when it fails. I've tried stubbing out all logic in the servlet and just returning successfully from my doGet method- with the same result. I've also noticed that at the same time as cron is failing tasks are queued up without processing for extended periods of time, even though no tasks are currently being executed. For an 1/2 hour here or there everything will seem OK, then cron will start to fail sporadically for another stretch, on all apps at the same time, so it doesn't appear to be specific to a single app. Example app is cliniconexdev. I can manually hit my cron and task handler URLs via browser over and over with no failures. App Engine status is currently showing Normal for everything.. /Tom -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[appengine-java] Re: 1.4.2: Can't make https call from dev server
JVM (On Windows 7), I've tried a few for the development server including: java version 1.6.0_22 Java(TM) SE Runtime Environment (build 1.6.0_22-b04) Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode) Using java.net.HttpURLConnection to connect to https://api.twilio.com and getting: javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate for: https://api.twilio.com api.twilio.com is only accessible over SSL so this problem is hampering development of our application with 1.4.2 /Tom On Monday, February 14, 2011 4:22:51 PM UTC-5, frew wrote: Hey everybody, In order to help me diagnose the problems, could you respond with what JDK (Platform, JDK manufacturer (e.g. Sun, OpenJDK), and version) you're running? As a temporary stopgap, if you're using URLFetch service, you can use the doNotValidateCertificate() method in FetchOptions (http://code.google.com/appengine/docs/java/javadoc/com/ google/appengine/api/urlfetch/FetchOptions.htmlhttp://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api/urlfetch/FetchOptions.html) to disable validation. -Fred On Feb 13, 5:02 am, harri mauria hma...@googlemail.com wrote: I am having similar problems with the recently released version com.google.appengine.eclipse.sdkbundle.1.4.2_1.4.2.v201102111811 Problems happen with the development server, not in production. I am performing Facebook login credentials authentication in the callback url. Interrupt stack is below: HTTP ERROR 500 Problem accessing /loginfacebookcallback. Reason: Could not verify SSL certificate for: https://graph.facebook.com/oauth/access_token?omitted Caused by: javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate for:https://graph.facebook.com/oauth/access_token?omitted at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc eption(URLFetchServiceImpl.java: 114) at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchService Impl.java: 41) at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler $Connection.fetchResponse(URLFetchServiceStreamHandler.java:418) at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler $Connection.getInputStream(URLFetchServiceStreamHandler.java:297) atwww.superpass.de.server.UrlFetcher.get(UrlFetcher.java:60) Following message is printed to the console: I/O exception (javax.net.ssl.SSLPeerUnverifiedException) caught when processing request: HTTPS hostname invalid: expected 'graph.facebook.com', received '*.facebook.com' Retrying request Regards, Harri On Feb 13, 11:27 am, Lior Harsat lior@gmail.com wrote: I am having a similar problem with https: I have been testing the SDK and I found one bug which seems critical. When I call the URLFetchService.fetch API with a https URL I get a javax.net.ssl.SSLHandshakeException. for example when I fetch https://spreadsheets.google.com/feeds/ list/ I get javax.net.ssl.SSLHandshakeException: Could not verify SSL certificate for:https://spreadsheets.google.com/feeds/list/... When I switch back to an older SDK this doesn't happen. Is there a Help us test Prerelease SDK 1.4.2 thread I can post this to? Can anyone confirm this bug reproduces? Thanx, Lior On Feb 12, 11:57 am, Peter Liu tiny...@gmail.com wrote: We are using a custom remote delegate to route datastore rpc call to live production test site on appspot. We did this because sometimes we want to develop with live data. However after upgrading to 1.4.2, https outbound call starts failing (see below). If we change to http, then it works again. It seems like some class is marked as restricted in this release. Is it intentional? What's the reason behind not allowing outbound https call? Thanks. java.lang.NoClassDefFoundError: javax.net.ssl.SSLHandshakeException is a restricted class. Please see the Google App Engine developer's guide for more details. at com.google.appengine.tools.development.agent.runtime.Runtime.reject(Runtime .java: 51) at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationExc eption(URLFetchServiceImpl.java: 114) at com.google.appengine.api.urlfetch.URLFetchServiceImpl.access $100(URLFetchServiceImpl.java:22) at com.google.appengine.api.urlfetch.URLFetchServiceImpl $1.convertException(URLFetchServiceImpl.java:86) at com.google.appengine.api.utils.FutureWrapper.get(FutureWrapper.java: 69) at com.pekca.main.delegate.RemoteDelegate$1.get(RemoteDelegate.java: 163) -- You received this message because you are
[google-appengine] Re: Memcache broken with Always On?
Through trial and error, I figured out that I was using it wrong. With
the JCache API, you have to make sure that you create and register
the cache once on each GAE instance (JVM) before you can just do a
getCache(String cachename) on it within that JVM.
So even in code that only reads from the cache, you need to make sure
the full cache initialization is performed or has already been done
somewhere in that JVM instance. For example:
cache = CacheManager.getInstance().getCache(testcache);
if (cache == null) {
cache = CacheManager.getInstance().getCacheFactory()
.createCache(Collections.emptyMap()); //Or whatever
properties are required
CacheManager.getInstance().registerCache(testcache, cache);
}
This is the same code you use to create and register the global cache
in the first place. For new instances, it appears to just create the
required link to the existing global cache. Makes sense to me now
since you don't know which instance or reader/writer code will be hit
the first time a cache comes into play.
After invoking the code above in my readers, I now see the cache and
entries in it being shared globally.
Thanks,
Tom
On Dec 15, 10:42 pm, Tom Phillips tphill0...@gmail.com wrote:
I posted on this this in the java forum, but no traction there and I
suspect it's not java specific.
Memcache entries seem to be being scoped to an instance, not global.
After a cache entry is made, it can be retrieved fine for any request
that goes to that same instance. Any request to one of the other
instances never finds the entry in the cache.
Bug in Always On? Bug in 1.4.0? Design intent?
Thanks,
Tom
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Memcache broken with Always On?
Hit Erik, From your stack trace, I think you are seeing something else. Have you got an explicit serialVersionUID set in the com.WikiHop.jdo.Pi class (your key is OK I think) that you are putting into Memcache? And if so has it changed at some point? It looks like you may be either trying to read out a different class entirely than was inserted into the cache, or have changed or added the serialVersionUID in the value class since an entry was inserted and it is now stale. serialVersionUID is a way of identifying a class (or version of a class) to the java serialization framework - if you change it you are telling it that it is effectively no longer compatible for serialization with previous instances, even if you have change nothing else in the class. So it's a good idea to set it early, and never change it for a class unless you have a good reason to invalidate older serialized entries. I'm not having any problems reading values out of the cache. The problem I'm seeing is that the entire entry can only be found if it was inserted by the same JVM (or instance') that inserted it. A scoping problem where it appears to be less global than it should be. You may want to start a separate thread on your issue if you are still seeing it and the above doesn't shed any light on it. Thanks, Tom On Dec 17, 4:11 am, Erik erik.e.wil...@gmail.com wrote: Missed some more errors: Caused by: java.io.InvalidClassException: com.WikiHop.jdo.Pi; local class incompatible: stream classdesc serialVersionUID = -6007756724690804603, local class serialVersionUID = -1398789866 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:579) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java: 1600) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java: 1513) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java: 1749) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at com.google.appengine.api.memcache.MemcacheSerialization.deserialize(MemcacheSerialization.java: 163) at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java: 283) ... 43 more On Dec 17, 3:56 am, Erik erik.e.wil...@gmail.com wrote: Not sure if it is related, but I was getting some interesting memcache errors that I had to work around: javax.servlet.ServletContext log: Exception while dispatching incoming RPC call com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public abstract java.lang.String com.WikiHop.client.SearchPath.find(java.lang.String,java.lang.String) throws java.lang.IllegalArgumentException' threw an unexpected exception: com.google.appengine.api.memcache.InvalidValueException: IO exception parsing value of '3434143' at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java: 378) ... Caused by: com.google.appengine.api.memcache.InvalidValueException: IO exception parsing value of '3434143' at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java: 289) at com.google.appengine.api.memcache.jsr107cache.GCache.get(GCache.java: 166) On Dec 16, 2:53 pm, Tom Phillips tphill0...@gmail.com wrote: Certainly Ikai, Firstly, some of my logs (url and classes slightly modified to hide some app details) that show how only the same instance picks up the entry. I'm determining and logging the instance using a UUID that I assign to a static class variable. The key for the cache lookup is passed in as a request query parameter to the followup requests. Here are two runs of the code to follow. In each, the bottom log is the cache entry being added. The next two are subsequent requests that both look for the entry. The entry is only found when then instance matches, regardless of which follow-up request. This first run shows how the bottom instance is different from the one that services both followup requests. So they don't find the entry: 50.16.21.20 - - [16/Dec/2010:11:19:40 -0800] GET /processg?key=619001 W 12-16 11:19AM 40.373 com.mycode.CallProcessGatherServlet doGet: LOOKING IN CACHE on Instance: b7c866f2-d74b-45df-9993-1d2c4705d3d1 W 12-16 11:19AM 40.373 com.mycode.CallProcessGatherServlet doGet: Message NOT found in cache for 619001 - building 50.16.21.20 - - [16/Dec/2010:11:19:35 -0800] POST /startreminder? key=619001pname=Optimal+Physio+Devatime=Friday+December+17%2C +2%3A18+PMrlang=enrvoice=woman W 12-16 11:19AM 33.173 com.mycode.CallMLServlet doGet: LOOKING IN CACHE on Instance: b7c866f2-d74b-45df-9993-1d2c4705d3d1 W 12-16 11:19AM 33.173 com.mycode.CallMLServlet doGet: Message NOT found in cache for 619001 - building 0.1.0.2 - - [16/Dec/2010:11:19:08 -0800] POST /tasks/voicereminder
[google-appengine] Re: Memcache broken with Always On?
Sorry, don't Hit Erik. Say Hi to him ;) On Dec 17, 10:46 am, Tom Phillips tphill0...@gmail.com wrote: Hit Erik, From your stack trace, I think you are seeing something else. Have you got an explicit serialVersionUID set in the com.WikiHop.jdo.Pi class (your key is OK I think) that you are putting into Memcache? And if so has it changed at some point? It looks like you may be either trying to read out a different class entirely than was inserted into the cache, or have changed or added the serialVersionUID in the value class since an entry was inserted and it is now stale. serialVersionUID is a way of identifying a class (or version of a class) to the java serialization framework - if you change it you are telling it that it is effectively no longer compatible for serialization with previous instances, even if you have change nothing else in the class. So it's a good idea to set it early, and never change it for a class unless you have a good reason to invalidate older serialized entries. I'm not having any problems reading values out of the cache. The problem I'm seeing is that the entire entry can only be found if it was inserted by the same JVM (or instance') that inserted it. A scoping problem where it appears to be less global than it should be. You may want to start a separate thread on your issue if you are still seeing it and the above doesn't shed any light on it. Thanks, Tom On Dec 17, 4:11 am, Erik erik.e.wil...@gmail.com wrote: Missed some more errors: Caused by: java.io.InvalidClassException: com.WikiHop.jdo.Pi; local class incompatible: stream classdesc serialVersionUID = -6007756724690804603, local class serialVersionUID = -1398789866 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:579) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java: 1600) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java: 1513) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java: 1749) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1346) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at com.google.appengine.api.memcache.MemcacheSerialization.deserialize(MemcacheSerialization.java: 163) at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java: 283) ... 43 more On Dec 17, 3:56 am, Erik erik.e.wil...@gmail.com wrote: Not sure if it is related, but I was getting some interesting memcache errors that I had to work around: javax.servlet.ServletContext log: Exception while dispatching incoming RPC call com.google.gwt.user.server.rpc.UnexpectedException: Service method 'public abstract java.lang.String com.WikiHop.client.SearchPath.find(java.lang.String,java.lang.String) throws java.lang.IllegalArgumentException' threw an unexpected exception: com.google.appengine.api.memcache.InvalidValueException: IO exception parsing value of '3434143' at com.google.gwt.user.server.rpc.RPC.encodeResponseForFailure(RPC.java: 378) ... Caused by: com.google.appengine.api.memcache.InvalidValueException: IO exception parsing value of '3434143' at com.google.appengine.api.memcache.MemcacheServiceImpl.get(MemcacheServiceImpl.java: 289) at com.google.appengine.api.memcache.jsr107cache.GCache.get(GCache.java: 166) On Dec 16, 2:53 pm, Tom Phillips tphill0...@gmail.com wrote: Certainly Ikai, Firstly, some of my logs (url and classes slightly modified to hide some app details) that show how only the same instance picks up the entry. I'm determining and logging the instance using a UUID that I assign to a static class variable. The key for the cache lookup is passed in as a request query parameter to the followup requests. Here are two runs of the code to follow. In each, the bottom log is the cache entry being added. The next two are subsequent requests that both look for the entry. The entry is only found when then instance matches, regardless of which follow-up request. This first run shows how the bottom instance is different from the one that services both followup requests. So they don't find the entry: 50.16.21.20 - - [16/Dec/2010:11:19:40 -0800] GET /processg?key=619001 W 12-16 11:19AM 40.373 com.mycode.CallProcessGatherServlet doGet: LOOKING IN CACHE on Instance: b7c866f2-d74b-45df-9993-1d2c4705d3d1 W 12-16 11:19AM 40.373 com.mycode.CallProcessGatherServlet doGet: Message NOT found in cache for 619001 - building 50.16.21.20 - - [16/Dec/2010:11:19:35 -0800] POST /startreminder? key=619001pname=Optimal+Physio+Devatime=Friday+December+17%2C +2%3A18+PMrlang=enrvoice=woman W 12-16 11:19AM 33.173 com.mycode.CallMLServlet doGet: LOOKING IN CACHE on Instance: b7c866f2-d74b
[google-appengine] Memcache broken with Always On?
I posted on this this in the java forum, but no traction there and I suspect it's not java specific. Memcache entries seem to be being scoped to an instance, not global. After a cache entry is made, it can be retrieved fine for any request that goes to that same instance. Any request to one of the other instances never finds the entry in the cache. Bug in Always On? Bug in 1.4.0? Design intent? Thanks, Tom -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Memcache broken with Always On?
) that retrieves the
entry from the cache:
Cache cache = null;
log.warning(LOOKING IN CACHE on Instance: +
TwilioPhoneAdapter.jvmId);
try {
cache =
CacheManager.getInstance().getCache(cache_reminderVoice);
if (cache != null) {
parts = (VoiceParts) cache.get(key);
}
if (parts != null) {
log.warning(Message for entry + key + FOUND in cache -
using);
}
} catch (final Exception e) {
log.warning(Cache cache_reminderVoice not yet created);
}
if (parts == null) {
log.warning(Message NOT found in cache for + key + -
building);
parts = ...//Builds the parts up from scratch
}
Thanks for checking it out Ikai.
Tom
On Dec 16, 1:32 pm, Ikai Lan (Google) ikai.l+gro...@google.com
wrote:
This is definitely not a design intent. Can you post any code or
reproduction steps?
--
Ikai Lan
Developer Programs Engineer, Google App Engine
Blogger:http://googleappengine.blogspot.com
Reddit:http://www.reddit.com/r/appengine
Twitter:http://twitter.com/app_engine
On Wed, Dec 15, 2010 at 7:42 PM, Tom Phillips tphill0...@gmail.com wrote:
I posted on this this in the java forum, but no traction there and I
suspect it's not java specific.
Memcache entries seem to be being scoped to an instance, not global.
After a cache entry is made, it can be retrieved fine for any request
that goes to that same instance. Any request to one of the other
instances never finds the entry in the cache.
Bug in Always On? Bug in 1.4.0? Design intent?
Thanks,
Tom
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.comgoogle-appengine%2bunsubscr...@googlegroups.com
.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[appengine-java] Re: Memcache entries - available how soon after entry?
So using UUID's class scope static variables I can now tell which JVM services each request. What I've confirmed is that at least when using reserved instances (Always On), Memcache entries are only found within the same JVM instance that created the cache entry. So I only get a cache hit if the next request goes to that same JVM. Is this expected behavior from Memcache, or should I raise a bug report? I pictured a highly-distributed memory cache available to the entire application. If it's per-JVM, I'm not sure why one wouldn't just use a Singleton or something to cache data, or why GAE RPC would be required when invoking the Memcache API. Thanks, Tom On Dec 14, 11:33 pm, Tom Phillips tphill0...@gmail.com wrote: I'm seeing unexpected (to me) Memcache behavior in my app ever since I enabled reserved instances (which I did first thing after 1.4.0). When one request adds a cache entry, subsequent requests anywhere from 10-60 seconds after are only able to find the entry sporadically. The cache is very lightly used and no expiry is set. I know that the correct key is being used by the writer and readers. The followup requests could be being handled by different JVM instances than the writer, but I wouldn't think this would make a difference. I'm not sure how to tell which instance services which request, I just now the request sequence. For all I know, the times it does find the cache entry might be only when serviced by the same JVM as the writer. Shouldn't Memcache entries be available to readers immediately after addition, and presumably available to all JVM instances? If not immediately for some reason, shouldn't they be available with 60 seconds? Thanks for any insight, Tom -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Memcache entries - available how soon after entry?
I'm seeing unexpected (to me) Memcache behavior in my app ever since I enabled reserved instances (which I did first thing after 1.4.0). When one request adds a cache entry, subsequent requests anywhere from 10-60 seconds after are only able to find the entry sporadically. The cache is very lightly used and no expiry is set. I know that the correct key is being used by the writer and readers. The followup requests could be being handled by different JVM instances than the writer, but I wouldn't think this would make a difference. I'm not sure how to tell which instance services which request, I just now the request sequence. For all I know, the times it does find the cache entry might be only when serviced by the same JVM as the writer. Shouldn't Memcache entries be available to readers immediately after addition, and presumably available to all JVM instances? If not immediately for some reason, shouldn't they be available with 60 seconds? Thanks for any insight, Tom -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Memcache misses after 1.4.0 update
In my application I use memcache to store some data for a subsequent request (usually issued within 10-20 seconds after the initial caching) that needs to respond as quickly as possible. Any delay greatly hurts the user experience as it results in a significant pause in an automated phone call while the the required data is rebuilt. I've got three app deployments with the same code base (at least the same in any way AFAICT that could matter for this). It seems that in two that I have redeployed since 1.4.0 came out (cliniconexdev,cliniconextest), the subsequent request consistently doesn't find the cached entry. In the version that hasn't been redeployed since 1.4.0 (cliniconexapp) the entry is found 100% of the time. On the development server, the caching still works fine and I get a hit every time. When I cache the data I have an immediate retrieval in the same thread that verifies that it is infact there, and in all cases it is found fine at that point. But for the two app instances with the problem, by the followup web request it appears to be gone from, or not yet available in, the cache. Traffic is light ATM on all app instances and memcache quotas are barely touched. I have billing and reserved instances enabled on all three which is really the only change I've made since 1.4.0 came out. Anyone else seen this or have any ideas? The only thing I can think of is that maybe new memcache entries aren't available to all jvm instances within that 10-20 seconds and I'm seeing it now because I have multiple reserved instances (at least on apps deployed since 1.4.0). If I can get that behavior confirmed I'll have to figure out how to work around it. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
