[google-appengine] GAE datastore insertion rate
I've developed a service on GAE, that takes small POSTS (160 bytes) from devices in the field, and stores them into the datastore. A device will send a package of 4 notes once a minute, during operational hours. These notes are unpacked and placed into the datastore (one entity Note). I started a simple benchmark of 1,000 devices. To my surprise, I got a bunch of these errors: "too much contention on these datastore entities. please try again." 1000 devices * 4 notes per minute gives me a rate of approximately 67 inserts per second. Is this an issue with the 'beta', or is this a reality of the datastore? Thank you, -David Story BTW, I was using approximately 10 CPU seconds per second during this run. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: GAE datastore insertion rate
After watching Brett's video, the answer became clear. http://sites.google.com/site/io/building-scalable-web-applications-with-google-app-engine Create a group entity for each device. Then all device's can have their notes inserted in parallel. I will give this a try and report back. -David Story --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: GAE datastore insertion rate
I created a root group entity for each Device. I modified the Note entity to use it's Device as its parent. device = Device.get_by_key_name(name) if device != None: for n in noteList: note = Note(parent=device, n) note.put() Now I'm seeing this error: The API call datastore_v3.Get() required more quota than is available. According to my interpretation of Brett, in that video, "get by key is very fast". Again I ask, is this a limit of the beta, or a reality of the datastore? -David Story --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: GAE datastore insertion rate
Thank you for your suggestions, I really appreciate them. My load test is sending data exactly as the physical devices in the field would do. Each device will send 4 Notes per POST, once a minute (a note every 15 seconds). I did not know about the "batch-put", I will make the necessary code changes for that. (Thank you Brett) I'm grouping the notes with the devices they came from. My first implementation, simply used the ReferenceProperty(), but quickly ran into the"too much contention for these entities" problem on the Note entity. Afterwatching Brett's presentation on "sharding", I thought I'd get moreparallelizm by making the devices a group entity and putting the notes in them. Which seemed to work, until I hit my quota. It's ok if the device group entity is 'locked' while inserting notes associated with it. Each physical device is unique, and it will only make one POST at a time. My goal is to be able to handle each device in parallel. If this works out, I'd like to run a 1 hour load test with 5,000 devices, to come up with a cost model for my company. I expect the number of devices to grow over the coming year, and need a scalable system to support this load. Thank you, -David Story --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: GAE datastore insertion rate
I got the contention errors, during my first run, using a ReferenceProperty to link a Note to a Device (not a parent). Only after I watched Brett's presentation did I add the device group entity and parent=device code. Adding the device group entity and parent=device code, seems to have solved the "too much contention for these entities" problem. Quote From 'Keys and Entity Groups': http://code.google.com/appengine/docs/python/datastore/keysandentitygroups.html#Entity_Groups_Ancestors_and_Paths "The more entity groups your application has—that is, the more root entities there are—the more efficiently the datastore can distribute the entity groups across datastore nodes." Brett does a great job of describing this in his presentation: http://sites.google.com/site/io/building-scalable-web-applications-with-google-app-engine For me, having all the notes from a device grouped together on a datastore node is great. I process the notes in sequence, but, I'd like to process the devices in parallel. This is where I'm looking for scalability, across the devices. I need to inform the App Engine that its ok to process these (notes) in sequence, but handle these (devices) in parallel. Making the device group entity seems to have done that. Thank you for your help, -David Story --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: GAE datastore insertion rate
Thank you Sebastian, I see what you are saying, and now wonder why it seems to work better with the devices as group entities. I'm attempting to delete the Note entities from the datastore, as I've reached 23%, and want to run another load test. I coded up a request to delete 512 Notes, then, make that request repeatedly. That presented me with this error: The API call datastore_v3.Delete() required more quota than is available. I slowed down the rate at which I make that request by sleeping a couple of seconds between each POST to the App Server. Same error. Then, using the dashboard, I attempt to view the Note entities and got a 500 Server error. Now, I can't view my Note entities. Funny thing is, when I look at the quota details page, and everything under the column on the far right (Rate), states "Okay". With only 3 of the Daily Quota's even showing a non-zero value. Request:CPU Time = 4% DataStore:Stored Data = 23% DataStore:Datastore CPU Time = 3% I'm trying some very simple things, and hope to get past these obstacles. In the cloud computing is a "good thing" and I want to take full advantage of it. Help, -David Story --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Server Error woes
I'm getting a "Server Error" message from the dashboard, when I go into the "Data Viewer" and select "TiwiNotes" (one of my data entities). I've had the data in the data store for a couple of months. I just purchased the "pay for higher quotas", and was getting ready to run another benchmark by deleting the existing entities. After deleting about 47000 items, my delete code (deleting 500 per request), threw an exception. I logged the exception, but it was blank. So, I attempted to look at the data via the "Data Viewer" and got the "Server Error". I know they are close, guess this is the fun part about being on the "bleeding edge". I'm cut! Medic! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Indexes have been building for more than 12 hours
Hi, I have two indexes that have been building for more than 12 hours. There are less than 50 items of that kind (TiwiNote), in my application (appid=tiwidev). Here are my indexes for that kind. The last two are the ones being built. - kind: TiwiNote ancestor: yes - kind: TiwiNote properties: - name: driverRef - name: nTime - kind: TiwiNote properties: - name: vehicleRef - name: nTime -David Story --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---