[appengine-java] Re: Unowned relations JPA examples for datanucleus plugin v2
A little more info on my example too. I'll refer you to my previous comment then. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Static DatastoreService and MemcacheService ?
Hi, I did a bit more research - It seems that it is not really necessary to have static Memcache and Datastore services as their instantiation is very fast: http://groups.google.com/group/google-appengine-java/browse_thread/thread/e19b792042b2ff9b And also we should not assume thread-safety of any class not documented as being thread-safe.. Also some testing of my own shows the runtime for creating 1000 instances: Datastore: 5 ms Memcache: 61 ms So I'll just create them whenever I need. Daniel -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/BgWMWoQytgQJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Unowned relations JPA examples for datanucleus plugin v2
On Feb 5, 3:14 am, datanucleus andy_jeffer...@yahoo.com wrote: A little more info on my example too. I'll refer you to my previous comment then. I've looked at your test example and am following the exact same pattern. You have me and somebody else here getting the exact same error. Something isn't working with this in a real GAE implementation outside of a test harness. Has anybody else been able to get this bidirectional m-n unowned relationship to work in a REAL environment? -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: Unowned relations JPA examples for datanucleus plugin v2
Ahhh...I had my side a and side b mixed up. My mappedBy attribute was on the wrong side causing the issue. :) It now works...thanks! -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] AppEngine JDO joint with GWT
Hi, scuse my bad english, I'm french. Since 3 days I try Google App Engine to deploy a GWT app. I use JDO to manage my entities. I can persist my data in the DataStore but when I try to read joins Entities are set to null. My joint is a OneToOne and I use the follow code to create my PrimaryKeys : @PrimaryKey @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY) @Extension(vendorName=datanucleus, key=gae.encoded-pk, value=true) I can get my joins entities when I add @Persistent(defaultFetchGroup = true) but console say's : The datastore does not support joins and therefore cannot honor requests to place related objects in the default fetch group. The field will be fetched lazily on first access. Can you help me to find a proper solution ? Thank in advance, Lexuor76. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/W0thDcWwlJwJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [google-appengine] Deleted App-Owner
Hello! Thank you for your answer. I only deleted the user. The app is still operating fine. But I don't have any possibility to manage the app (statistics, limits, etc). appengine.google.com Just gives me the Create new application option ;( I will try to contact google once more... They pretty ignored me for now ;( thank you, stefan Am Samstag, 4. Februar 2012 08:14:28 UTC+1 schrieb Robert Kluin: Hey Stefan, Did you delete the app too, or just the admin user? I seem to recall that such apps get periodically garbage collected, but I'm not 100% positive about that. You might file an issue asking for help, or possibly a billing issue. I'm not sure which would be a better choice. http://code.google.com/p/googleappengine/issues/entry http://code.google.com/appengine/kb/billing.html Robert On Fri, Feb 3, 2012 at 06:48, swalkner swalk...@ylog.at wrote: Hello, I created an app (shared-groups) with a user I later deleted... How can I register a new user for this app? Thank you, stefan -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/dXJ8oVfqU-0J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/0k1rXO6OG68J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Running Out of Database Read Operations
David, you might want to rethink the design of your app and use push updates instead of pulling: http://code.google.com/appengine/docs/python/channel/overview.html http://code.google.com/appengine/docs/java/channel/overview.html Also, caching queries results will help you a lot. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/9WVje6plTXkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] python unittests + indexes
I bumped into this while unittesting and then running my app with dev_appserver/production: unittests go fine but later I discover I don't have required indexes (I'm using dev_appserver.py ... --require_indexes) So, I figured I'd do something like this in my setUp(): def setUp(self): ... self.testbed = testbed.Testbed() self.testbed.activate() self.testbed.init_datastore_v3_stub(require_indexes=True) dev_appserver_index.SetupIndexes(None, %s/../ % os.path.dirname(__file__)) I'm setting up datastore stub with require_indexes=True and later: dev_appserver_index.SetupIndexes(None, %s/../ % os.path.dirname(__file__)) - None tells it to use default app_id (whatever it is set to in app.yaml) - second arg is the path to app root (I have my tests in app_root/tests dir) Just posting this in case someone else finds it useful. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/7SSy_C8IoE4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] gae(web2py) + twitter bootstrap 2.0
service like reddit(korean) : http://www.feed9.com/ source : https://github.com/sungchi/feed9 Thank you web2py, Thank you twitter ! most prominent changes of twitter bootstrap 2.0 : New 12-column, responsive grid system New table styles with a common base class for improved compatibility with other tools New form styles with smarter defaults, requiring less HTML Icons, graciously provided by Glyphicons New, smarter navigation components New buttons, button groups, and button dropdowns New, simpler alert messages New javascript plug-ins like carousel, collapse, and typeahead http://twitter.github.com/bootstrap/ -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Running Out of Database Read Operations
Thanks for the advice. But, won't channels limit me to 100 users per day? And does memcache pickup changes to the database quickly? On Feb 5, 2012 5:26 AM, alex a...@cloudware.it wrote: David, you might want to rethink the design of your app and use push updates instead of pulling: http://code.google.com/appengine/docs/python/channel/overview.html http://code.google.com/appengine/docs/java/channel/overview.html Also, caching queries results will help you a lot. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/9WVje6plTXkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: SLOW
http://goo.gl/Mmhzo 2012-02-05 06:28:04.398 /feed/ 200 90894ms 41kb Apple-PubSub/65.23 I'd file a production issue, but Google ignores those anyway. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a pushtask) that would lease atask, or tasks, from the pullqueue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using URL Fetch to pull data this might let you do it in parallel without increasing your costs much (if any). Robert On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote: Here are logs from three consecutivetaskexecutions over the past weekend, with only identifying information removed. You'll see that eachtask completes in a few milliseconds, but are 20 seconds apart (remember: I've already checked myqueueconfigurations, nothing else is running on this backend, and I later solved the problem by setting countdown=1 when adding thetask). I don't see any pending latency mentioned. 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304913211 instance=0 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 instance=0 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 instance=0 The 20 seconds seems to happen regardless of length oftask. Even though my tasks mostly complete in a couple minutes, I do have cases where they take several minutes, and I don't see a difference. Of course, when atasktakes 5-10 minutes to complete, I'm going to notice and care about a 20-second delaymuch less than when I'm trying to spin through a few tasks in a minute (which is a real-world need for me as well). When reading up on pull queues a while back, I was a little confused about where I would use them with my own backends. I definitely could see an application for offloading work to an AWS Linux instance. But in either case, could you explain why it might help? I saw you mention in a separate thread how M/S can perform differently from HRD even in cases where one wouldn't expect to see a difference. When I get around to it I'm going to create a tiny HRD app and run the same tests through that. I also wonder if M/S could be responsible for frequent latencies in my admin console. Those have gotten more frequent and annoying the past couple of months ... -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] hola
Ciao, the filesystem on appengine is read-only ^^. Cheers On Sat, Feb 4, 2012 at 8:58 PM, Mjrosse rosse0...@centromedicohermanngmeiner.com wrote: hello everyone. can somebody explain me how to fix this problem with my aplication on appspot. the issue is this, I upload a proyect made in php with java and the page run perfect but, if i go to admin panel i get this problem The settings file was found at includes/settings.php but is not writable. Please set the appropriate permissions to make the settings file writable. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Alessandro Aglietti webLog http://blog.aqquadro.it - realtimeCVhttp://cv.alessandroaglietti.com - Google+ https://alessandroaglietti.com * * *Gli'è ora d'illuminassi, tirassi sui i calzoni e levassi* -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
In my case, since I was getting the 20-second delay almost 100% of the time, setting countdown=1 was the answer. If you only see it happen every 20 or more request then of course it won't help. In my case I also run all tasks on the backend. They're slightly more expensive per hour than frontends (due merely to the lower number of free hours) but in my case I more than make up for it with the fact that I have full control on the number of requests that will spin up, and I need to be able to control that number separately for tasks vs. users hitting my site. On Feb 5, 11:24 am, Carter jcmas...@gmail.com wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a pushtask) that would lease atask, or tasks, from the pullqueue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using URL Fetch to pull data this might let you do it in parallel without increasing your costs much (if any). Robert On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote: Here are logs from three consecutivetaskexecutions over the past weekend, with only identifying information removed. You'll see that eachtask completes in a few milliseconds, but are 20 seconds apart (remember: I've already checked myqueueconfigurations, nothing else is running on this backend, and I later solved the problem by setting countdown=1 when adding thetask). I don't see any pending latency mentioned. 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304913211 instance=0 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 instance=0 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 instance=0 The 20 seconds seems to happen regardless of length oftask. Even though my tasks mostly complete in a couple minutes, I do have cases where they take several minutes, and I don't see a difference. Of course, when atasktakes 5-10 minutes to complete, I'm going to notice and care about a 20-second delaymuch less than when I'm trying to spin through a few tasks in a minute (which is a real-world need for me as well). When reading up on pull queues a while back, I was a little confused about where I would use them with my own backends. I definitely could see an application for offloading work to an AWS Linux instance. But in either case, could you explain why it might help? I saw you mention in a separate thread how M/S can perform differently from HRD even in cases where one wouldn't expect to see a difference. When I get around to it I'm going to create a tiny HRD app and run the same tests through that. I also wonder if M/S could be responsible for frequent latencies in my admin console. Those have gotten more frequent and annoying the past couple of months ... -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
that I have full control on the number of requests that will spin up, err, number of instances that will spin up, rather ... On Feb 5, 11:30 am, Dave Loomer dloo...@gmail.com wrote: In my case, since I was getting the 20-second delay almost 100% of the time, setting countdown=1 was the answer. If you only see it happen every 20 or more request then of course it won't help. In my case I also run all tasks on the backend. They're slightly more expensive per hour than frontends (due merely to the lower number of free hours) but in my case I more than make up for it with the fact that I have full control on the number of requests that will spin up, and I need to be able to control that number separately for tasks vs. users hitting my site. On Feb 5, 11:24 am, Carter jcmas...@gmail.com wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a pushtask) that would lease atask, or tasks, from the pullqueue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using URL Fetch to pull data this might let you do it in parallel without increasing your costs much (if any). Robert On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote: Here are logs from three consecutivetaskexecutions over the past weekend, with only identifying information removed. You'll see that eachtask completes in a few milliseconds, but are 20 seconds apart (remember: I've already checked myqueueconfigurations, nothing else is running on this backend, and I later solved the problem by setting countdown=1 when adding thetask). I don't see any pending latency mentioned. 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304913211 instance=0 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 instance=0 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 instance=0 The 20 seconds seems to happen regardless of length oftask. Even though my tasks mostly complete in a couple minutes, I do have cases where they take several minutes, and I don't see a difference. Of course, when atasktakes 5-10 minutes to complete, I'm going to notice and care about a 20-second delaymuch less than when I'm trying to spin through a few tasks in a minute (which is a real-world need for me as well). When reading up on pull queues a while back, I was a little confused about where I would use them with my own backends. I definitely could see an application for offloading work to an AWS Linux instance. But in either case, could you explain why it might help? I saw you mention in a separate thread how M/S can perform differently from HRD even in cases where one wouldn't expect to see a difference. When I get around to it I'm going to create a tiny HRD app and run the same tests through that. I also wonder if M/S could be responsible for frequent latencies in my admin console. Those have gotten more frequent and annoying the past couple of months ... -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine
Re: [google-appengine] Re: Question about Data Store Read operations
Hi Robert, I got it! I have not considered that query offset also causes small operation. thank you very much Robert for your input. On Wed, Feb 1, 2012 at 9:00 AM, Robert Kluin robert.kl...@gmail.com wrote: Hey Andrew, So you need to remember that running a query will result in many reads (the number of results, plus a scan iirc) -- not just one. That could well be the source of your problem. As for why memcache isn't being effective, that's harder to say; that could be a coding issue or a memcache issue. Robert On Tue, Jan 31, 2012 at 07:18, Andrew Osipenko dev7...@gmail.com wrote: Hi Robert, Thanks for you being interested to help me. Let me tell my long sad story: I have Java appengine app. The app has only one page. This page performs search with various parameters, fetches 11 entities and displays search result. My app has 3400 page views per day and each day it exceeds it's datastore read quota. Lets count: (11 entities + 1 query) * 3400 = 41800 41800 is nearly 5 (I have not counted other pages) This looks ok. After this I added memcache with refresh interval = 1 hour. But I am still getting datastore read quota exceeding. Appstats reports the following data: user.CreateLoginURL 674datastore_v3.RunQuery 311memcache.Get 309 memcache.Set 150datastore_v3.Put 34datastore_v3.Get 20 blobstore.CreateUploadURL 14datastore_v3.Delete 8mail.Send 3 Appstats data is not clear to me. According to docs it describes recent 1000 requests or so. user.CreateLoginURL is called on each page, so appstats describe 674 page requests datastore_v3.RunQuery=311 But why memcache.Set = 150? It should be = 311= datastore_v3.RunQuery = cache miss count Perhaps appstats is not the app which can provide exact numbers. My question is: do you have an idea why my app still get's datastore read quota exceeding even after memcache introducing? -thanks, Andrew On Tue, Jan 31, 2012 at 7:51 AM, Robert Kluin robert.kl...@gmail.comwrote: Hey Andrew, What symptoms are you seeing exactly? Robert On Mon, Jan 30, 2012 at 05:48, Andrew Osipenko dev7...@gmail.com wrote: Hi Robert, no, session disabling is the first thing I did to reduce datastore access count. See http://code.google.com/p/rent-map/source/browse/trunk/trunk/src/main/webapp/WEB-INF/appengine-web.xml Second thing I did was memcache introducing. So I used all standard datastore friendly technics. On Mon, Jan 30, 2012 at 7:33 AM, Robert Kluin robert.kl...@gmail.com wrote: Are you using sessions? On Sun, Jan 29, 2012 at 05:37, Andrew Osipenko dev7...@gmail.com wrote: Hi Frank, Have you solved it? I have nearly the same problem. I think that appstats do not count some datastore read operations. I thought my application accessed datastore during intialization before appstat filter is being invoked. But I can't find anything bad during my app initialization. Do you have any ideas? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Tr6-P0a_XWAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com . To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group
[google-appengine] I hate google checkout
Every time I try to pay my appengine bill I get struck by another bug Usually authentication related bugs, where you need to login logoff delete cookies etc Sometimes payment related bugs, for example in order to update a credit card you need to delete it and add it again just wanted to express how pissed off I am -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/IG2jkmpKVPAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Carter wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. Been a burr under the saddle forever. What I really don't understand -- assuming GAE engineers never see the benefit of providing at least one priority/reliability queue -- is why the heck there is never any explanation about how tasks get scheduled, and why these weird delays happen. It is either: 1) If we told you we would have to shoot you, or 2) We can't see the benefit of you understanding this. -stevep On Feb 5, 9:24 am, Carter jcmas...@gmail.com wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a pushtask) that would lease atask, or tasks, from the pullqueue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using URL Fetch to pull data this might let you do it in parallel without increasing your costs much (if any). Robert On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote: Here are logs from three consecutivetaskexecutions over the past weekend, with only identifying information removed. You'll see that eachtask completes in a few milliseconds, but are 20 seconds apart (remember: I've already checked myqueueconfigurations, nothing else is running on this backend, and I later solved the problem by setting countdown=1 when adding thetask). I don't see any pending latency mentioned. 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304913211 instance=0 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 instance=0 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 instance=0 The 20 seconds seems to happen regardless of length oftask. Even though my tasks mostly complete in a couple minutes, I do have cases where they take several minutes, and I don't see a difference. Of course, when atasktakes 5-10 minutes to complete, I'm going to notice and care about a 20-second delaymuch less than when I'm trying to spin through a few tasks in a minute (which is a real-world need for me as well). When reading up on pull queues a while back, I was a little confused about where I would use them with my own backends. I definitely could see an application for offloading work to an AWS Linux instance. But in either case, could you explain why it might help? I saw you mention in a separate thread how M/S can perform differently from HRD even in cases where one wouldn't expect to see a difference. When I get around to it I'm going to create a tiny HRD app and run the same tests through that. I also wonder if M/S could be responsible for frequent latencies in my admin console. Those have gotten more frequent and annoying the past couple of months ... -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/lbNQRQdSx0AJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
RE: [google-appengine] I hate google checkout
As someone who uses Adwords API, and randomly gets bills with no email, and not account ID be very scared of the alternative. Google doesn't seem to know how to work with outside companies for money exchange. Adwords, Adsense, are awful. Apps For Domains is worse. API Services are so bad that I have considered using a brokers and paying almost 2x as much to be sure which client I am billing. The really big problem is that there isn't a method to easily reconcile which client a bill goes with, so we can't scale our offering because we couldn't easily do accounting. I definitely feel your pain. Brandon Wirtz BlackWaterOps: President / Lead Mercenary Description: http://www.linkedin.com/img/signature/bg_slate_385x42.jpg Work: 510-992-6548 Toll Free: 866-400-4536 IM: drak...@gmail.com (Google Talk) Skype: drakegreene YouTube: http://www.youtube.com/blackwateropsdotcom BlackWaterOpsDotCom http://www.blackwaterops.com/ BlackWater Ops http://www.cloudonastring.com/ Cloud On A String Mastermind Group -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. image003.jpg
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
We would have no need to shoot anyone. However, the explanations quickly become obsolete. They enter the folklore in the form that was current at the time and become entrenched as incorrect information when the implementations have changed. Task Queues use best effort scheduling. They're not real time all the time, although when our best efforts are running smoothly they can appear real time. For scheduling, the task eta marks the earliest time at which the task can run. We can't guarantee that a task WILL run at that time. Steve, we're interested to know about the 10-20 minute delays you've seen. Can you tell us the app id, queue, and whether the tasks were added transactionally? An example from your logs would be very helpful. Nick Verne On Mon, Feb 6, 2012 at 9:27 AM, stevep prosse...@gmail.com wrote: Carter wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. Been a burr under the saddle forever. What I really don't understand -- assuming GAE engineers never see the benefit of providing at least one priority/reliability queue -- is why the heck there is never any explanation about how tasks get scheduled, and why these weird delays happen. It is either: 1) If we told you we would have to shoot you, or 2) We can't see the benefit of you understanding this. -stevep On Feb 5, 9:24 am, Carter jcmas...@gmail.com wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a pushtask) that would lease atask, or tasks, from the pullqueue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using URL Fetch to pull data this might let you do it in parallel without increasing your costs much (if any). Robert On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote: Here are logs from three consecutivetaskexecutions over the past weekend, with only identifying information removed. You'll see that eachtask completes in a few milliseconds, but are 20 seconds apart (remember: I've already checked myqueueconfigurations, nothing else is running on this backend, and I later solved the problem by setting countdown=1 when adding thetask). I don't see any pending latency mentioned. 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304913211 instance=0 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 instance=0 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 instance=0 The 20 seconds seems to happen regardless of length oftask. Even though my tasks mostly complete in a couple minutes, I do have cases where they take several minutes, and I don't see a difference. Of course, when atasktakes 5-10 minutes to complete, I'm going to notice and care about a 20-second delaymuch less than when I'm trying to spin through a few tasks in a minute (which is a real-world need for me as well). When reading up on pull queues a while back, I was a little confused about where I would use them with my own backends. I definitely could see an application for offloading work to an AWS Linux instance. But in either case, could you explain why it might help? I saw you mention in a separate thread how M/S can perform differently from HRD even in cases where one wouldn't expect to see a difference. When I get around to it I'm going to create a tiny HRD app and run the same tests through that. I also wonder if M/S could be responsible for frequent latencies in my admin console. Those have gotten more frequent and annoying the past couple of months ... -- You received this message because you are subscribed to the Google Groups Google App Engine group.
[google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
As the OP you may be interested in my app ID as well: mn-live. I provided some logs a few posts back and some exhaustive details at the beginning. However, you won't see this issue popping up anymore on my app since I solved it by setting countdown=1 a week ago. Since then, tasks start very reliably after a 1.5 second delay. If I remove the countdown parameter, then it returns to 20 seconds (+/- .01) pretty reliably. On Feb 5, 5:59 pm, Nicholas Verne nve...@google.com wrote: We would have no need to shoot anyone. However, the explanations quickly become obsolete. They enter the folklore in the form that was current at the time and become entrenched as incorrect information when the implementations have changed. Task Queues use best effort scheduling. They're not real time all the time, although when our best efforts are running smoothly they can appear real time. For scheduling, the task eta marks the earliest time at which the task can run. We can't guarantee that a task WILL run at that time. Steve, we're interested to know about the 10-20 minute delays you've seen. Can you tell us the app id, queue, and whether the tasks were added transactionally? An example from your logs would be very helpful. Nick Verne On Mon, Feb 6, 2012 at 9:27 AM, stevep prosse...@gmail.com wrote: Carter wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. Been a burr under the saddle forever. What I really don't understand -- assuming GAE engineers never see the benefit of providing at least one priority/reliability queue -- is why the heck there is never any explanation about how tasks get scheduled, and why these weird delays happen. It is either: 1) If we told you we would have to shoot you, or 2) We can't see the benefit of you understanding this. -stevep On Feb 5, 9:24 am, Carter jcmas...@gmail.com wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a pushtask) that would lease atask, or tasks, from the pullqueue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using URL Fetch to pull data this might let you do it in parallel without increasing your costs much (if any). Robert On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote: Here are logs from three consecutivetaskexecutions over the past weekend, with only identifying information removed. You'll see that eachtask completes in a few milliseconds, but are 20 seconds apart (remember: I've already checked myqueueconfigurations, nothing else is running on this backend, and I later solved the problem by setting countdown=1 when adding thetask). I don't see any pending latency mentioned. 0.1.0.2 - - [27/Jan/2012:18:33:20 -0800] 200 124 ms=10 cpu_ms=47 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304913211 instance=0 0.1.0.2 - - [27/Jan/2012:18:33:00 -0800] 200 124 ms=11 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=15804554889304912461 instance=0 0.1.0.2 - - [27/Jan/2012:18:32:41 -0800] 200 124 ms=26 cpu_ms=0 api_cpu_ms=0 cpm_usd=0.60 queue_name=overnight-tasks task_name=4499136807998063691 instance=0 The 20 seconds seems to happen regardless of length oftask. Even though my tasks mostly complete in a couple minutes, I do have cases where they take several minutes, and I don't see a difference. Of course, when atasktakes 5-10 minutes to complete, I'm going to notice and care about a 20-second delaymuch less than when I'm trying to spin through a few tasks in a minute (which is a real-world need for me as well). When reading up on pull queues a while back, I was a little confused about where I would use them with my own backends. I definitely could see an application for offloading work to an AWS Linux instance. But in either
[google-appengine] Remove post from mail-archive.com
Hi, I have contacted mail-archive.com to delete a post I have posted but they told me that I need to send a request to you. Could you please assist with what information do you need to remove the post from - http://www.mail-archive.com Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Huge spike in ms/request since last week
Lately, I have seen GAE taking much, much longer to process requests than it did just a week ago. Nothing changed in my code, but GAE now is taking 4000-12000ms to respond to requests. What makes is worse is that I have plenty of instances available with 0 requests on them. Has anyone else seen this happen? What can I do to fix it?I have gone as far as to spin up 15 extra instances (and paid through the nose for them) but nothing seems to send requests to the other idle instances reliably. My bill has gone from 70-90c/day to $5-8/day without any code change or increase in traffic. In fact, I am losing traffic because of the huge latency. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: SLOW
This is starting to cost me a lot, but in payments to Google and lost customers due to the high latency being seen by everyone. Is google working on this? On Feb 1, 10:15 am, A1programmer derrick.simp...@gmail.com wrote: Getting the same issues here... again... Java for me. Requests that are usually sub 200ms are now 3 to 4 seconds (if I'm lucky). Occasionally they're between 20 and 30 seconds. On Feb 1, 12:15 pm, pdknsk pdk...@googlemail.com wrote: It hasn't helped here. The problem isn't even the short 10s spikes, but the sometimes many hours before it, with 1s response time (from usually ~100ms). It seems quite clear to me that the response time builds up slowly, suddenly peaks, and returns back to normal immediately. Shortly after that it builds up again. http://goo.gl/uUXqD -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] High latency and huge increase in ms/request
Lately, I have seen GAE taking much, much longer to process requests than it did just a week ago. Nothing changed in my code, but GAE now is taking 4000-12000ms to respond to requests. What makes is worse is that I have plenty of instances available with 0 requests on them. Has anyone else seen this happen? What can I do to fix it?I have gone as far as to spin up 15 extra instances (and paid through the nose for them) but nothing seems to send requests to the other idle instances reliably. My bill has gone from 70-90c/day to $5-8/day without any code change or increase in traffic. In fact, I am losing traffic because of the huge latency. QPS*Latency*RequestsErrors Age Memory Availability 0.000 0.0 ms 13780 10:10:0957.9 MBytes Dynamic 0.000 0.0 ms 16810 15:39:5757.2 MBytes Dynamic 0.017 9687.0 ms 886 0 10:19:1056.7 MBytes Dynamic -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Remove post
Hi, I have contacted mail-archive.com to delete a post I have posted but they told me that I need to send a request to you first. Could you please assist with what information I need to send to remove the post from - http://www.mail-archive.com Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Remove post
Hi, I have contacted mail-archive.com to delete a post I have posted but they told me that I need to send a request to you. Could you please assist with what information do you need to remove the post from - http://www.mail-archive.com Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: SLOW
I opened a case issue #6871, please star in the hopes that Google will notice... http://code.google.com/p/googleappengine/issues/detail?id=6871 On Feb 5, 7:49 am, ssg sun...@gmail.com wrote: This is starting to cost me a lot, but in payments to Google and lost customers due to the high latency being seen by everyone. Is google working on this? On Feb 1, 10:15 am, A1programmer derrick.simp...@gmail.com wrote: Getting the same issues here... again... Java for me. Requests that are usually sub 200ms are now 3 to 4 seconds (if I'm lucky). Occasionally they're between 20 and 30 seconds. On Feb 1, 12:15 pm, pdknsk pdk...@googlemail.com wrote: It hasn't helped here. The problem isn't even the short 10s spikes, but the sometimes many hours before it, with 1s response time (from usually ~100ms). It seems quite clear to me that the response time builds up slowly, suddenly peaks, and returns back to normal immediately. Shortly after that it builds up again. http://goo.gl/uUXqD -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Huge spike in ms/request since last week
https://docs.google.com/document/d/13W_1WVZHP-2xDrJ5mMigEqM9W5Xbw9sNgXiDUTU2F5s/edit On Feb 5, 7:36 am, ssg sun...@gmail.com wrote: Lately, I have seen GAE taking much, much longer to process requests than it did just a week ago. Nothing changed in my code, but GAE now is taking 4000-12000ms to respond to requests. What makes is worse is that I have plenty of instances available with 0 requests on them. Has anyone else seen this happen? What can I do to fix it?I have gone as far as to spin up 15 extra instances (and paid through the nose for them) but nothing seems to send requests to the other idle instances reliably. My bill has gone from 70-90c/day to $5-8/day without any code change or increase in traffic. In fact, I am losing traffic because of the huge latency. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] file.create() throws an ApiDeadlineExceededException very frequently
Hi Gaurav, I wonder if this issue is related to how your code works. In the past I've used the files api to write a lot of blobs, via tasks, simultaneously without issues. If you're able to put together some simple code and instructions to replicate the issue with that little load, someone can spot the issue. You might also consider filing an issue. Another idea to experiment with is adjusting the RPC deadline. If you reduce it, you'll get the timeout fast which will allow you to retry more times. In some cases, this can help. Robert On Sun, Feb 5, 2012 at 00:24, Gaurav Misra gau...@vurbal.com wrote: Hi Robert, I tried that solution and it has helped reduce the rate but with 4 test users we are still seeing a very high rate for these errors. I try three times to make it work but it still fails. I think if I try more than a few times, I might hit the request timeout deadline, and its not good for the user to wait a long time either. Thanks for your help, Gaurav -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/dQ3ipcgw7F4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Remove post from mail-archive.com
You should probably include a link to the post in Google Groups. On Fri, Feb 3, 2012 at 23:02, Jade Elizabeth jade.elizabet...@yahoo.com wrote: Hi, I have contacted mail-archive.com to delete a post I have posted but they told that I need to send a request to you. Could you please assist with what information do you need to remove the post from - http://www.mail-archive.com Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Frontend instance hours being consumed with zero requests
Hi Troy, Do you have cron jobs or tasks that run on that app? Have you reduced the max idle instances to 1? Robert On Sat, Feb 4, 2012 at 14:35, Troy tde...@gmail.com wrote: I have an app that is basically a test version of another app. So everything is setup the same and uses the same code, but it's not accessible by our users, only the internal team. We have not done any testing or usage of the test version as of late, so it's not serving any request. Yet when looking in the Dashboard, we see our Frontend Instance Hours being used. Can anyone explain this? We're almost at half a day's free quota, yet it's served no requests! Thanks for any help! -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Remove Archive post
Please don't keep posting the same request. You need to provide the link to Google Groups post you want removed. On Sat, Feb 4, 2012 at 23:19, Jadeli jade.elizabet...@yahoo.com wrote: Hi, I would like to remove an archive post from mail-archive.com and I need instruction to do this. mail-archive.com won't remove the post from they system until a request has been sent by Google App engine Admin/owner. Please help, thanks -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Running Out of Database Read Operations
Hey David, A query counts as one read plus one read per entity returned. The costs of various operation are detailed on the billings page: http://code.google.com/appengine/docs/billing.html You probably need to come up with a way to filter your queries so they only return what is needed. Or, come up with some other way to make the reads more efficient. There are a number of ways you could achieve this, but without knowing more about what you're doing it is hard to suggest something. Most likely, you'd need (or want) to sync memcache with the datastore yourself. To quickly test how much this would help you could cache the query results, then clear the cache each time something is added. It is simplistic, but it might give you an idea. Alex's suggestion about the channel API is also good. The free quotas are sufficient for small apps, but for anything actually using resources they won't stretch very far. Robert On Sun, Feb 5, 2012 at 00:45, David davidbdu...@gmail.com wrote: My application (studyvolved) was sending 3 post requests a second to constantly update a user's page with the appropriate information. I managed to succeed my 50k read operations in 13 successfully served requests by 1 instance. This took between 30 seconds and 2 minutes. I am reworking the app to only change on user refresh but I would eventually like to return to the constantly updating model. Does anyone have suggestions for how to do live updates without using too many resources? What constitutes a read? Is a .gql() call more than one read? Also, would my app's inefficiencies have shown up when I was hosting it on my computer? It seemed fine then. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Are there any slick methods for Owned Collection item orphan removal?
Hey Brandon, Iterate over the _keys_ making sure the parent entity exists. You could design it to prevent rechecking for the same parent over and over, which will help reduce costs. You can iterate over a lot of entities pretty fast with a keys-only query. Insert one task per X parents you need to check. You'll need to do some empirical testing, but I'd start with 500 or 1000, then do a single batch get for them. If any aren't found, remove the corresponding orphans. If you expect a lot of orphans, I'd use smaller groups. Robert On Sun, Feb 5, 2012 at 11:49, Brandon Donnelson branflake2...@gmail.com wrote: Are there any slick datastore methods for Owned Collection orphan removal? Brandon Donnelson http://c.gwt-examples.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/FvITFUJIbMcJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] hola
You might find the reference useful when debugging these types of problems: http://code.google.com/appengine/docs/java/runtime.html#The_Sandbox Robert On Sun, Feb 5, 2012 at 12:28, Alessandro Aglietti alessandro.aglie...@gmail.com wrote: Ciao, the filesystem on appengine is read-only ^^. Cheers On Sat, Feb 4, 2012 at 8:58 PM, Mjrosse rosse0...@centromedicohermanngmeiner.com wrote: hello everyone. can somebody explain me how to fix this problem with my aplication on appspot. the issue is this, I upload a proyect made in php with java and the page run perfect but, if i go to admin panel i get this problem The settings file was found at includes/settings.php but is not writable. Please set the appropriate permissions to make the settings file writable. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- Alessandro Aglietti webLog - realtimeCV - Google+ Gli'è ora d'illuminassi, tirassi sui i calzoni e levassi -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
That's interesting. Did the queue sit there for a long time not running anything, or running tasks very slowly? Are the tasks in that queue generally long-running? I _very_ infrequently bump into that type of issue, but I periodically will see one queue slow down for a while. It *seems* to happen far more often in queues with slower tasks, but I don't have any recent empirical evidence of that. And I *think* I've been told that should not be the case. Robert On Sun, Feb 5, 2012 at 19:27, Carter Maslan car...@maslan.com wrote: Nicholas - For our examples of the 10-20 minute delay: app_id=s~camiologger queue=image-label (but several other queues experience the same long delays sometimes: content-process, counter-update, etc...) The tasks were not added with transactions; just this code: Queue queueP = QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH); TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL) .param(key, contentKeyString) .method(TaskOptions.Method.GET)); Let me know if you need more info. We noticed this in the last few weeks. Carter On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer dloo...@gmail.com wrote: As the OP you may be interested in my app ID as well: mn-live. I provided some logs a few posts back and some exhaustive details at the beginning. However, you won't see this issue popping up anymore on my app since I solved it by setting countdown=1 a week ago. Since then, tasks start very reliably after a 1.5 second delay. If I remove the countdown parameter, then it returns to 20 seconds (+/- .01) pretty reliably. On Feb 5, 5:59 pm, Nicholas Verne nve...@google.com wrote: We would have no need to shoot anyone. However, the explanations quickly become obsolete. They enter the folklore in the form that was current at the time and become entrenched as incorrect information when the implementations have changed. Task Queues use best effort scheduling. They're not real time all the time, although when our best efforts are running smoothly they can appear real time. For scheduling, the task eta marks the earliest time at which the task can run. We can't guarantee that a task WILL run at that time. Steve, we're interested to know about the 10-20 minute delays you've seen. Can you tell us the app id, queue, and whether the tasks were added transactionally? An example from your logs would be very helpful. Nick Verne On Mon, Feb 6, 2012 at 9:27 AM, stevep prosse...@gmail.com wrote: Carter wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. Been a burr under the saddle forever. What I really don't understand -- assuming GAE engineers never see the benefit of providing at least one priority/reliability queue -- is why the heck there is never any explanation about how tasks get scheduled, and why these weird delays happen. It is either: 1) If we told you we would have to shoot you, or 2) We can't see the benefit of you understanding this. -stevep On Feb 5, 9:24 am, Carter jcmas...@gmail.com wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a pushtask) that would lease atask, or tasks, from the pullqueue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using URL Fetch to pull data this might let you do it in parallel without increasing your costs much (if any). Robert On Wed, Feb 1, 2012 at 14:25, Dave Loomer dloo...@gmail.com wrote: Here are logs from three consecutivetaskexecutions over the past weekend, with only identifying information removed. You'll see that eachtask completes in a few milliseconds, but are 20 seconds apart (remember: I've already checked myqueueconfigurations, nothing else is running on this backend, and I later solved the problem by setting countdown=1 when adding
Re: [google-appengine] Remove post
Time for someone to block this person as a spammer -- yikes. On Sun, Feb 5, 2012 at 18:45, Jade Elizabeth jade.elizabet...@yahoo.com wrote: Hi, I have contacted mail-archive.com to delete a post I have posted but they told me that I need to send a request to you. Could you please assist with what information do you need to remove the post from - http://www.mail-archive.com Thanks. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Huge spike in ms/request since last week
Hi, Is your app master-slave or high replication datastore? What do the requests that are taking longer do? Do they make a URL fetch call to another site, access the datastore, or memcahce, etc...? Are you seeing more loading requests than normal? Do you see higher than usual pending latencies in your logs? Robert On Sun, Feb 5, 2012 at 20:58, ssg sun...@gmail.com wrote: https://docs.google.com/document/d/13W_1WVZHP-2xDrJ5mMigEqM9W5Xbw9sNgXiDUTU2F5s/edit On Feb 5, 7:36 am, ssg sun...@gmail.com wrote: Lately, I have seen GAE taking much, much longer to process requests than it did just a week ago. Nothing changed in my code, but GAE now is taking 4000-12000ms to respond to requests. What makes is worse is that I have plenty of instances available with 0 requests on them. Has anyone else seen this happen? What can I do to fix it?I have gone as far as to spin up 15 extra instances (and paid through the nose for them) but nothing seems to send requests to the other idle instances reliably. My bill has gone from 70-90c/day to $5-8/day without any code change or increase in traffic. In fact, I am losing traffic because of the huge latency. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: SLOW
Are you making a URL fetch call in that? On Sun, Feb 5, 2012 at 11:42, pdknsk pdk...@googlemail.com wrote: http://goo.gl/Mmhzo 2012-02-05 06:28:04.398 /feed/ 200 90894ms 41kb Apple-PubSub/65.23 I'd file a production issue, but Google ignores those anyway. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
I Just looked at the last 80 that ran. That queue's tasks are running in between 19ms and 2486ms with most of them running around 28ms. The variability relates to the number of quadtree searches needed, but other queues that experience similar delays have running time without much variation(e.g. predictable counter updates) When the delays happen, there just aren't many tasks in the queue at all. It appears that the delayed tasks are just sitting in the queue idle. On Feb 5, 2012, at 9:17 PM, Robert Kluin robert.kl...@gmail.com wrote: That's interesting. Did the queue sit there for a long time not running anything, or running tasks very slowly? Are the tasks in that queue generally long-running? I _very_ infrequently bump into that type of issue, but I periodically will see one queue slow down for a while. It *seems* to happen far more often in queues with slower tasks, but I don't have any recent empirical evidence of that. And I *think* I've been told that should not be the case. Robert On Sun, Feb 5, 2012 at 19:27, Carter Maslan car...@maslan.com wrote: Nicholas - For our examples of the 10-20 minute delay: app_id=s~camiologger queue=image-label (but several other queues experience the same long delays sometimes: content-process, counter-update, etc...) The tasks were not added with transactions; just this code: Queue queueP = QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH); TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL) .param(key, contentKeyString) .method(TaskOptions.Method.GET)); Let me know if you need more info. We noticed this in the last few weeks. Carter On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer dloo...@gmail.com wrote: As the OP you may be interested in my app ID as well: mn-live. I provided some logs a few posts back and some exhaustive details at the beginning. However, you won't see this issue popping up anymore on my app since I solved it by setting countdown=1 a week ago. Since then, tasks start very reliably after a 1.5 second delay. If I remove the countdown parameter, then it returns to 20 seconds (+/- .01) pretty reliably. On Feb 5, 5:59 pm, Nicholas Verne nve...@google.com wrote: We would have no need to shoot anyone. However, the explanations quickly become obsolete. They enter the folklore in the form that was current at the time and become entrenched as incorrect information when the implementations have changed. Task Queues use best effort scheduling. They're not real time all the time, although when our best efforts are running smoothly they can appear real time. For scheduling, the task eta marks the earliest time at which the task can run. We can't guarantee that a task WILL run at that time. Steve, we're interested to know about the 10-20 minute delays you've seen. Can you tell us the app id, queue, and whether the tasks were added transactionally? An example from your logs would be very helpful. Nick Verne On Mon, Feb 6, 2012 at 9:27 AM, stevep prosse...@gmail.com wrote: Carter wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. Been a burr under the saddle forever. What I really don't understand -- assuming GAE engineers never see the benefit of providing at least one priority/reliability queue -- is why the heck there is never any explanation about how tasks get scheduled, and why these weird delays happen. It is either: 1) If we told you we would have to shoot you, or 2) We can't see the benefit of you understanding this. -stevep On Feb 5, 9:24 am, Carter jcmas...@gmail.com wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a pushtask) that would lease atask, or tasks, from the pullqueue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using URL Fetch to pull data this might let you do it in parallel without increasing your costs much (if any). Robert On Wed, Feb 1,
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
Does the app get a lot of front-end traffic or is it sitting idle when the delays occur? On Mon, Feb 6, 2012 at 01:38, Carter Maslan jcmas...@gmail.com wrote: I Just looked at the last 80 that ran. That queue's tasks are running in between 19ms and 2486ms with most of them running around 28ms. The variability relates to the number of quadtree searches needed, but other queues that experience similar delays have running time without much variation(e.g. predictable counter updates) When the delays happen, there just aren't many tasks in the queue at all. It appears that the delayed tasks are just sitting in the queue idle. On Feb 5, 2012, at 9:17 PM, Robert Kluin robert.kl...@gmail.com wrote: That's interesting. Did the queue sit there for a long time not running anything, or running tasks very slowly? Are the tasks in that queue generally long-running? I _very_ infrequently bump into that type of issue, but I periodically will see one queue slow down for a while. It *seems* to happen far more often in queues with slower tasks, but I don't have any recent empirical evidence of that. And I *think* I've been told that should not be the case. Robert On Sun, Feb 5, 2012 at 19:27, Carter Maslan car...@maslan.com wrote: Nicholas - For our examples of the 10-20 minute delay: app_id=s~camiologger queue=image-label (but several other queues experience the same long delays sometimes: content-process, counter-update, etc...) The tasks were not added with transactions; just this code: Queue queueP = QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH); TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL) .param(key, contentKeyString) .method(TaskOptions.Method.GET)); Let me know if you need more info. We noticed this in the last few weeks. Carter On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer dloo...@gmail.com wrote: As the OP you may be interested in my app ID as well: mn-live. I provided some logs a few posts back and some exhaustive details at the beginning. However, you won't see this issue popping up anymore on my app since I solved it by setting countdown=1 a week ago. Since then, tasks start very reliably after a 1.5 second delay. If I remove the countdown parameter, then it returns to 20 seconds (+/- .01) pretty reliably. On Feb 5, 5:59 pm, Nicholas Verne nve...@google.com wrote: We would have no need to shoot anyone. However, the explanations quickly become obsolete. They enter the folklore in the form that was current at the time and become entrenched as incorrect information when the implementations have changed. Task Queues use best effort scheduling. They're not real time all the time, although when our best efforts are running smoothly they can appear real time. For scheduling, the task eta marks the earliest time at which the task can run. We can't guarantee that a task WILL run at that time. Steve, we're interested to know about the 10-20 minute delays you've seen. Can you tell us the app id, queue, and whether the tasks were added transactionally? An example from your logs would be very helpful. Nick Verne On Mon, Feb 6, 2012 at 9:27 AM, stevep prosse...@gmail.com wrote: Carter wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. Been a burr under the saddle forever. What I really don't understand -- assuming GAE engineers never see the benefit of providing at least one priority/reliability queue -- is why the heck there is never any explanation about how tasks get scheduled, and why these weird delays happen. It is either: 1) If we told you we would have to shoot you, or 2) We can't see the benefit of you understanding this. -stevep On Feb 5, 9:24 am, Carter jcmas...@gmail.com wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks and do the work in parallel. Basically you'd have a backend that ran a loop (possibly initiated via a pushtask) that would lease atask, or tasks, from the pullqueue, do the work, delete those tasks, then repeat from the lease stage. The cool thing is that if you're, for example, using
Re: [google-appengine] Re: (mostly) Consistent 20-second delay in starting backend tasks
The app always has front end traffic and I have noticed the delay at times with high traffic. On Feb 5, 2012 10:57 PM, Robert Kluin robert.kl...@gmail.com wrote: Does the app get a lot of front-end traffic or is it sitting idle when the delays occur? On Mon, Feb 6, 2012 at 01:38, Carter Maslan jcmas...@gmail.com wrote: I Just looked at the last 80 that ran. That queue's tasks are running in between 19ms and 2486ms with most of them running around 28ms. The variability relates to the number of quadtree searches needed, but other queues that experience similar delays have running time without much variation(e.g. predictable counter updates) When the delays happen, there just aren't many tasks in the queue at all. It appears that the delayed tasks are just sitting in the queue idle. On Feb 5, 2012, at 9:17 PM, Robert Kluin robert.kl...@gmail.com wrote: That's interesting. Did the queue sit there for a long time not running anything, or running tasks very slowly? Are the tasks in that queue generally long-running? I _very_ infrequently bump into that type of issue, but I periodically will see one queue slow down for a while. It *seems* to happen far more often in queues with slower tasks, but I don't have any recent empirical evidence of that. And I *think* I've been told that should not be the case. Robert On Sun, Feb 5, 2012 at 19:27, Carter Maslan car...@maslan.com wrote: Nicholas - For our examples of the 10-20 minute delay: app_id=s~camiologger queue=image-label (but several other queues experience the same long delays sometimes: content-process, counter-update, etc...) The tasks were not added with transactions; just this code: Queue queueP = QueueFactory.getQueue(ServerUtils.QUEUE_NAME_IMAGE_LABEL_PUSH); TaskHandle th = queueP.add(withUrl(ServerUtils.PATH_ADMIN_MOTION_LABEL) .param(key, contentKeyString) .method(TaskOptions.Method.GET)); Let me know if you need more info. We noticed this in the last few weeks. Carter On Sun, Feb 5, 2012 at 4:05 PM, Dave Loomer dloo...@gmail.com wrote: As the OP you may be interested in my app ID as well: mn-live. I provided some logs a few posts back and some exhaustive details at the beginning. However, you won't see this issue popping up anymore on my app since I solved it by setting countdown=1 a week ago. Since then, tasks start very reliably after a 1.5 second delay. If I remove the countdown parameter, then it returns to 20 seconds (+/- .01) pretty reliably. On Feb 5, 5:59 pm, Nicholas Verne nve...@google.com wrote: We would have no need to shoot anyone. However, the explanations quickly become obsolete. They enter the folklore in the form that was current at the time and become entrenched as incorrect information when the implementations have changed. Task Queues use best effort scheduling. They're not real time all the time, although when our best efforts are running smoothly they can appear real time. For scheduling, the task eta marks the earliest time at which the task can run. We can't guarantee that a task WILL run at that time. Steve, we're interested to know about the 10-20 minute delays you've seen. Can you tell us the app id, queue, and whether the tasks were added transactionally? An example from your logs would be very helpful. Nick Verne On Mon, Feb 6, 2012 at 9:27 AM, stevep prosse...@gmail.com wrote: Carter wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. Been a burr under the saddle forever. What I really don't understand -- assuming GAE engineers never see the benefit of providing at least one priority/reliability queue -- is why the heck there is never any explanation about how tasks get scheduled, and why these weird delays happen. It is either: 1) If we told you we would have to shoot you, or 2) We can't see the benefit of you understanding this. -stevep On Feb 5, 9:24 am, Carter jcmas...@gmail.com wrote: We regularly but erratically see 10-20 minute delays in running push queue tasks. The tasks sit in the queue with ETA as high as 20 minutes *ago* without any errors or retries. (the problem seems unrelated to queue settings since our Maximum Rate, Enorced Rate and Maximum Concurrent all far exceed the queue's throughput at the time of the delays) Any tips or clues on how to prevent this while still using push queues without backends? On Feb 1, 9:03 pm, Robert Kluin robert.kl...@gmail.com wrote: Hey Dave, Hopefully Nick will be able to offer some insight into the cause of your issues. I'd guess it is something related to having very few tasks (one) in thequeue, and it not getting scheduled rapidly. In your case, you could use pull queues to immediately fetch the nexttaskwhen finished with atask. Or even to fetch multiple tasks
