[google-appengine] Indexes have been building for more than 12 hours
Hi, I have two indexes that have been building for more than 12 hours. There are less than 50 items of that kind (TiwiNote), in my application (appid=tiwidev). Here are my indexes for that kind. The last two are the ones being built. - kind: TiwiNote ancestor: yes - kind: TiwiNote properties: - name: driverRef - name: nTime - kind: TiwiNote properties: - name: vehicleRef - name: nTime -David Story --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Server Error woes
I'm getting a Server Error message from the dashboard, when I go into the Data Viewer and select TiwiNotes (one of my data entities). I've had the data in the data store for a couple of months. I just purchased the pay for higher quotas, and was getting ready to run another benchmark by deleting the existing entities. After deleting about 47000 items, my delete code (deleting 500 per request), threw an exception. I logged the exception, but it was blank. So, I attempted to look at the data via the Data Viewer and got the Server Error. I know they are close, guess this is the fun part about being on the bleeding edge. I'm cut! Medic! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: GAE datastore insertion rate
Thank you for your suggestions, I really appreciate them. My load test is sending data exactly as the physical devices in the field would do. Each device will send 4 Notes per POST, once a minute (a note every 15 seconds). I did not know about the batch-put, I will make the necessary code changes for that. (Thank you Brett) I'm grouping the notes with the devices they came from. My first implementation, simply used the ReferenceProperty(), but quickly ran into thetoo much contention for these entities problem on the Note entity. Afterwatching Brett's presentation on sharding, I thought I'd get moreparallelizm by making the devices a group entity and putting the notes in them. Which seemed to work, until I hit my quota. It's ok if the device group entity is 'locked' while inserting notes associated with it. Each physical device is unique, and it will only make one POST at a time. My goal is to be able to handle each device in parallel. If this works out, I'd like to run a 1 hour load test with 5,000 devices, to come up with a cost model for my company. I expect the number of devices to grow over the coming year, and need a scalable system to support this load. Thank you, -David Story --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: GAE datastore insertion rate
I got the contention errors, during my first run, using a ReferenceProperty to link a Note to a Device (not a parent). Only after I watched Brett's presentation did I add the device group entity and parent=device code. Adding the device group entity and parent=device code, seems to have solved the too much contention for these entities problem. Quote From 'Keys and Entity Groups': http://code.google.com/appengine/docs/python/datastore/keysandentitygroups.html#Entity_Groups_Ancestors_and_Paths The more entity groups your application has—that is, the more root entities there are—the more efficiently the datastore can distribute the entity groups across datastore nodes. Brett does a great job of describing this in his presentation: http://sites.google.com/site/io/building-scalable-web-applications-with-google-app-engine For me, having all the notes from a device grouped together on a datastore node is great. I process the notes in sequence, but, I'd like to process the devices in parallel. This is where I'm looking for scalability, across the devices. I need to inform the App Engine that its ok to process these (notes) in sequence, but handle these (devices) in parallel. Making the device group entity seems to have done that. Thank you for your help, -David Story --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] GAE datastore insertion rate
I've developed a service on GAE, that takes small POSTS (160 bytes) from devices in the field, and stores them into the datastore. A device will send a package of 4 notes once a minute, during operational hours. These notes are unpacked and placed into the datastore (one entity Note). I started a simple benchmark of 1,000 devices. To my surprise, I got a bunch of these errors: too much contention on these datastore entities. please try again. 1000 devices * 4 notes per minute gives me a rate of approximately 67 inserts per second. Is this an issue with the 'beta', or is this a reality of the datastore? Thank you, -David Story BTW, I was using approximately 10 CPU seconds per second during this run. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
