[appengine-java] HELP: My app is getting lots of HardDeadlineExceededError today.
Application was running ok yesturday. Still had about 10% HardDeadlineExceededError, but the same task that executed yesturday are not failing almost 100% of the time right now. I even cut back the amount of records it is processing from 100 to 10 and it is still failing. How do I get ahold of someone at Google, to see if there is something wrong with the server my app is on or something. Why would the same proccess consume more CPU time one day than another? -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: HELP: My app is getting lots of HardDeadlineExceededError today.
Further explanation. I had a large increase in users which caused a chain reaction. Tasks that used to take about 10-20 seconds started taking longer and using more CPU time. Interestingly, these tasks are doing the exact same thing, but under a higher load, they take longer and use more CPU time. I can understand the process taking longer under load, but why would it use a significant amount more of CPU time? I will admit that my code is not the most optimal, but then again isn't the main focus of GAE built in scalability and speed? In short, I needed on the fly scalability and speed with an influx of new users and I don't feel like I got that with GAE? Worse, there was no way for me to contact someone who could help me out. I'm left with the feeling that GAE does not do a good job of scaling up and it is kind of slow as well. Creating and updating records seems painfully expensive, time out due to new instance creation happens way to often. Now this is fine for a free app, but I'm going way over my daily free quota and it is not cheap. Maybe I should be using Phython and not Java? On Aug 6, 10:52 pm, SC2PlayerFinder wrote: > Application was running ok yesturday. Still had about 10% > HardDeadlineExceededError, but the same task that executed yesturday > are not failing almost 100% of the time right now. I even cut back > the amount of records it is processing from 100 to 10 and it is still > failing. > > How do I get ahold of someone at Google, to see if there is something > wrong with the server my app is on or something. Why would the same > proccess consume more CPU time one day than another? -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: HELP: My app is getting lots of HardDeadlineExceededError today.
Thanks for the response. I wanted to get back to you sooner, but I have been scrambling to optimize the application. I'm making some progress, however, it is still a relative CPU hog due to an exploding index and the frequent updates the application needs to make in order to keep the data current. 71000 CPU to save 100 records seems rather expensive?. My next step is to run appstat on it and see what is going on. Current version of the app is running a sc2pf6.appstat.com. The older version is running at sc2pf2.appspot.com. On Aug 9, 3:51 pm, "Ikai L (Google)" wrote: > I've got a few questions for you: > > 1. What is your application ID? > 2. What are you doing in the tasks? > 3. Are you using transactions or entity groups? > 4. Are you writing to the same datastore entities? > > My guess is that you are running into write contention in the datastore. > It'd be very helpful to understand what it is you're doing. > > App Engine will scale up the number of instances if you are returning web > requests in less than 800ms (thought 400ms is optimal), but if you are > writing to a small number of entities over and over it won't be able to > resolve that bottleneck. > > On Mon, Aug 9, 2010 at 12:28 PM, SC2PlayerFinder > wrote: > > > > > > > > > Further explanation. I had a large increase in users which caused a > > chain reaction. Tasks that used to take about 10-20 seconds started > > taking longer and using more CPU time. Interestingly, these tasks are > > doing the exact same thing, but under a higher load, they take longer > > and use more CPU time. I can understand the process taking longer > > under load, but why would it use a significant amount more of CPU > > time? I will admit that my code is not the most optimal, but then > > again isn't the main focus of GAE built in scalability and speed? In > > short, I needed on the fly scalability and speed with an influx of new > > users and I don't feel like I got that with GAE? Worse, there was no > > way for me to contact someone who could help me out. > > > I'm left with the feeling that GAE does not do a good job of scaling > > up and it is kind of slow as well. Creating and updating records > > seems painfully expensive, time out due to new instance creation > > happens way to often. Now this is fine for a free app, but I'm going > > way over my daily free quota and it is not cheap. Maybe I should be > > using Phython and not Java? > > > On Aug 6, 10:52 pm, SC2PlayerFinder wrote: > > > Application was running ok yesturday. Still had about 10% > > > HardDeadlineExceededError, but the same task that executed yesturday > > > are not failing almost 100% of the time right now. I even cut back > > > the amount of records it is processing from 100 to 10 and it is still > > > failing. > > > > How do I get ahold of someone at Google, to see if there is something > > > wrong with the server my app is on or something. Why would the same > > > proccess consume more CPU time one day than another? > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine for Java" group. > > To post to this group, send email to > > google-appengine-j...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine-java+unsubscr...@googlegroups.com > unsubscr...@googlegroups.com> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine-java?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: HELP: My app is getting lots of HardDeadlineExceededError today.
Used appstat and didn't notice anything out of the ordinary. Still thinking that the main issue here is exploding index and I'm not sure how to get around that other than to cut search features from the site, but that would more or less make the app not very useful. On Aug 12, 10:19 pm, SC2PlayerFinder wrote: > Thanks for the response. I wanted to get back to you sooner, but I > have been scrambling to optimize the application. I'm making some > progress, however, it is still a relative CPU hog due to an exploding > index and the frequent updates the application needs to make in order > to keep the data current. 71000 CPU to save 100 records seems rather > expensive?. > > My next step is to run appstat on it and see what is going on. > > Current version of the app is running a sc2pf6.appstat.com. The older > version is running at sc2pf2.appspot.com. > > On Aug 9, 3:51 pm, "Ikai L (Google)" wrote: > > > > > I've got a few questions for you: > > > 1. What is your application ID? > > 2. What are you doing in the tasks? > > 3. Are you using transactions or entity groups? > > 4. Are you writing to the same datastore entities? > > > My guess is that you are running into write contention in the datastore. > > It'd be very helpful to understand what it is you're doing. > > > App Engine will scale up the number of instances if you are returning web > > requests in less than 800ms (thought 400ms is optimal), but if you are > > writing to a small number of entities over and over it won't be able to > > resolve that bottleneck. > > > On Mon, Aug 9, 2010 at 12:28 PM, SC2PlayerFinder > > wrote: > > > > Further explanation. I had a large increase in users which caused a > > > chain reaction. Tasks that used to take about 10-20 seconds started > > > taking longer and using more CPU time. Interestingly, these tasks are > > > doing the exact same thing, but under a higher load, they take longer > > > and use more CPU time. I can understand the process taking longer > > > under load, but why would it use a significant amount more of CPU > > > time? I will admit that my code is not the most optimal, but then > > > again isn't the main focus of GAE built in scalability and speed? In > > > short, I needed on the fly scalability and speed with an influx of new > > > users and I don't feel like I got that with GAE? Worse, there was no > > > way for me to contact someone who could help me out. > > > > I'm left with the feeling that GAE does not do a good job of scaling > > > up and it is kind of slow as well. Creating and updating records > > > seems painfully expensive, time out due to new instance creation > > > happens way to often. Now this is fine for a free app, but I'm going > > > way over my daily free quota and it is not cheap. Maybe I should be > > > using Phython and not Java? > > > > On Aug 6, 10:52 pm, SC2PlayerFinder wrote: > > > > Application was running ok yesturday. Still had about 10% > > > > HardDeadlineExceededError, but the same task that executed yesturday > > > > are not failing almost 100% of the time right now. I even cut back > > > > the amount of records it is processing from 100 to 10 and it is still > > > > failing. > > > > > How do I get ahold of someone at Google, to see if there is something > > > > wrong with the server my app is on or something. Why would the same > > > > proccess consume more CPU time one day than another? > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Google App Engine for Java" group. > > > To post to this group, send email to > > > google-appengine-j...@googlegroups.com. > > > To unsubscribe from this group, send email to > > > google-appengine-java+unsubscr...@googlegroups.com > > unsubscr...@googlegroups.com> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine-java?hl=en. > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Engine > > Blog:http://googleappengine.blogspot.com > > Twitter:http://twitter.com/app_engine > > Reddit:http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: HELP: My app is getting lots of HardDeadlineExceededError today.
2 Main parts of the application. Font End Display StarCraft 2 Player 1v1 Ladder information from all Regions. Search options: Order by (6 categories) filter by (Region) filter by (Race). <-- This is the cause of my exploding index. Display 100 players per page. I have implemented some cache use for this part, so it is not much of an issue currently. Back End (This is the big CPU Hog) Load Player information from battle.net http://us.battle.net/sc2/en/profile/388538/1/HuK/ladder/1794#current-rank This is done by a task that gets the data, parses it and saves any player changes. There is 100 players per page, but usually less than 30 need to be updated each scan. Each save/update takes almost 1 second. .5 seconds with a version that has NO INDEXES. Currently the speed is not an issue, but the cost of CPU time for Datastore operations is. Ideally, I would like to support the almost 1 million players on battle.net like sc2Ranks.com But that would not be practical cost wise using the Datastore. For now, I'm only supporting the top 10K players. Don't get me wrong. I really love GAE. I have created a few small apps that run fine. However, this app is looking like it won't be practical for GAE due to the Datastore? On Aug 13, 1:24 pm, "Ikai L (Google)" wrote: > Saving 100 entities in a single request, especially if a TON of indexes will > be created is very expensive and will cause a lot of CPU to be consumed. > There probably is a design that will allow you to accomplish what you need > to do without this limitation. Can you post your use case? > > On Thu, Aug 12, 2010 at 11:02 PM, SC2PlayerFinder > > > > > > wrote: > > > Used appstat and didn't notice anything out of the ordinary. Still > > thinking that the main issue here is exploding index and I'm not sure > > how to get around that other than to cut search features from the > > site, but that would more or less make the app not very useful. > > > On Aug 12, 10:19 pm, SC2PlayerFinder > > wrote: > > > Thanks for the response. I wanted to get back to you sooner, but I > > > have been scrambling to optimize the application. I'm making some > > > progress, however, it is still a relative CPU hog due to an exploding > > > index and the frequent updates the application needs to make in order > > > to keep the data current. 71000 CPU to save 100 records seems rather > > > expensive?. > > > > My next step is to run appstat on it and see what is going on. > > > > Current version of the app is running a sc2pf6.appstat.com. The older > > > version is running at sc2pf2.appspot.com. > > > > On Aug 9, 3:51 pm, "Ikai L (Google)" wrote: > > > > > I've got a few questions for you: > > > > > 1. What is your application ID? > > > > 2. What are you doing in the tasks? > > > > 3. Are you using transactions or entity groups? > > > > 4. Are you writing to the same datastore entities? > > > > > My guess is that you are running into write contention in the > > datastore. > > > > It'd be very helpful to understand what it is you're doing. > > > > > App Engine will scale up the number of instances if you are returning > > web > > > > requests in less than 800ms (thought 400ms is optimal), but if you are > > > > writing to a small number of entities over and over it won't be able to > > > > resolve that bottleneck. > > > > > On Mon, Aug 9, 2010 at 12:28 PM, SC2PlayerFinder > > > > wrote: > > > > > > Further explanation. I had a large increase in users which caused a > > > > > chain reaction. Tasks that used to take about 10-20 seconds started > > > > > taking longer and using more CPU time. Interestingly, these tasks are > > > > > doing the exact same thing, but under a higher load, they take longer > > > > > and use more CPU time. I can understand the process taking longer > > > > > under load, but why would it use a significant amount more of CPU > > > > > time? I will admit that my code is not the most optimal, but then > > > > > again isn't the main focus of GAE built in scalability and speed? In > > > > > short, I needed on the fly scalability and speed with an influx of > > new > > > > > users and I don't feel like I got that with GAE? Worse, there was no > > > > > way for me to contact someone who could help me out. > > > > > > I'm left with the feeling that GAE does not do a good job of scaling > > > > > up
[appengine-java] offset past 1000 does not work 1.3.6
Version 1.3.6 - August 17, 2010¶ Results of datastore count() queries and offsets for all datastore queries are no longer capped at 1000. Thought this was working past 1000 when I tested it the other day, but now it will continue to return the same results after 1000. query.setRange(1000, 1100) will return the same thing as query.setRange(900,1000). -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
