[google-appengine] Recent massive increase in URLFetch DownloadError: ApplicationError: 2
Hi there, We have been experiencing a massive increase in URLFetch errors in the couple of weeks. Our application used to run perfectly with no errors but recent connections have had frequent issues which impacts our customers (these are connections to a merchant API). The only error we get from URLFetch is a generic DownloadError: ApplicationError: 2. Are these timeouts or another production issue? Is there someone at Google that can have a look to see what is actually happening with the connections and give a better diagnosis? I have already approached the other side of the connection who assures me they are seeing no problems on their end. Thanks, Nikolaj -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/YTlYBIVuXnwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Possibility of a strange memcache.decr() failure?
Hi all, I had a logic failure in my application this morning (4:30am UTC) that I can only explain if memcache.decr() returned an undecremented value (this is on the Python SDK). I am not 100% sure that this actually happened, but it's my current best diagnosis. So I wanted to ask the Google folks if this is even remotely possible or if None would definitely be returned instead? Thanks, Nikolaj -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/I1M-wBj3QvwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: How to throttle or batch transactional updates to an entity group
Steve, I thought I had gotten everything out of that talk but clearly I was sleeping in class! The memcache fork join queue looks very interesting - I'm definitely giving that a try. Nikolaj On Sep 23, 1:00 am, Steve Sherrie st...@wasteofpaper.com wrote: Definitely checkout Brett Slatkin's Building high-throughput data pipelines with Google App Engine http://www.google.com/events/io/2010/sessions/high-throughput-data-pi... I/O Session which should give you a fundamental understanding of the caveats involved with batch writing. You have to do a little work to make sure writes are not added to a batch that's already applied (or is otherwise invalid), but Brett explains how to do that with memcache locking. Steve On 11-09-22 07:11 PM, Brandon Wirtz wrote: Start a another datastore that doesn't use the same entity, but uses an incremental system. On Write: In a table with format Writes to do, Entity to Overwrite, Data for Entity 1,Brandon Wirtz, +$120 2,Brandon Wirtz, -$120 3,Brandon Wirtz, +$0.66 Increment Write to do count, and store transaction. If a Read request for the entity comes in: Get Entity Get Latest committed Write To Do Value Check Write Cache Recalculate Value based on Entity Value and All Changes On Batch: For batch size calculate all values, and update entities Store Late todo value as Last committed write. You can now batch as often as every 2 seconds and not hit the entity write limit. -Original Message- From: google-appengine@googlegroups.com [mailto:google-appengine@googlegroups.com] On Behalf Of Nikolaj Sent: Thursday, September 22, 2011 3:59 PM To: Google App Engine Subject: [google-appengine] How to throttle or batch transactional updates to an entity group Hi there, I'm developing an application that requires me to create lots of child entities within the same entity group transactionally. The tasks being generated to do this come in unpredictable bursts - sometimes 100/s for an entity group. This is clearly exceeding the 1/s limit for transaction contention and causing some real problems for my poor application. I need a way to either batch the required updates for each entity group so I can run through them serially or otherwise schedule each update to run on a 1/s basis. The ideas I've come up with so far: 1. Use some kind of memcache'd timestamp or counter to schedule or throttle the updates. My question is how to do this in a way that will work with 100's of tasks hungrily looking for their time slot? Should I do a CAS for when the last task was scheduled (would this even work with the contention we're talking about?) or can I use an atomic operation like incr() somehow? 2. Store the work items somewhere and then at some later time query for the work items for an entity group and create the children in serial. I'd like to avoid this, if possible, as it creates many other headaches. My application is already using the taskqueue to great effect. This really seems like a queue problem - if only there was some kind of dynamic subqueue where I could add tasks with a 'namespace' of the entity group. Then entity groups could run in parallel, but the entity group items would still be in serial. Does something like that exist? Any suggestions or experiences on how I can work around this would be appreciated. Many thanks, Nikolaj -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] How to throttle or batch transactional updates to an entity group
Hi there, I'm developing an application that requires me to create lots of child entities within the same entity group transactionally. The tasks being generated to do this come in unpredictable bursts - sometimes 100/s for an entity group. This is clearly exceeding the 1/s limit for transaction contention and causing some real problems for my poor application. I need a way to either batch the required updates for each entity group so I can run through them serially or otherwise schedule each update to run on a 1/s basis. The ideas I've come up with so far: 1. Use some kind of memcache'd timestamp or counter to schedule or throttle the updates. My question is how to do this in a way that will work with 100's of tasks hungrily looking for their time slot? Should I do a CAS for when the last task was scheduled (would this even work with the contention we're talking about?) or can I use an atomic operation like incr() somehow? 2. Store the work items somewhere and then at some later time query for the work items for an entity group and create the children in serial. I'd like to avoid this, if possible, as it creates many other headaches. My application is already using the taskqueue to great effect. This really seems like a queue problem - if only there was some kind of dynamic subqueue where I could add tasks with a 'namespace' of the entity group. Then entity groups could run in parallel, but the entity group items would still be in serial. Does something like that exist? Any suggestions or experiences on how I can work around this would be appreciated. Many thanks, Nikolaj -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] NoseGAE not updating index.yaml
Hi there, I was wondering if it is possible to configure or modify the NoseGAE plugin to automatically update the index.yaml file when queries are run in tests. I have 100% test coverage so this would be quite useful. Has anybody had any luck with such a testing suite setup? Nikolaj -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Cannot log in. Not authorized to access this application
It seems the web app at google sometimes gets confused, when you are logged in to more than one (google) account at a time. You log in OK (authentication), but when you are trying to access the GAE page (authorization) I think the webapp somehow does that with the wrong cookie. For me it worked to use another browser / delete all cookies related to google, and then try again. If this tip was written on the authorization denied page, I could have saved an hour of pulling out my hair :-) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
