[google-appengine] IN queries
I read somewhere that IN queries are processed serially by the datastore. GOOGers, what is a rough rule-of-thumb on the benefit of using IN? For example, if the base RT latency for anything with the datastore is Nms, then could guesstimate that using N for a list of 3 is not a huge latency win, but using IN for 10 is. Does using IN substantially cut down on the api_cpu_ms used vs sequential queries? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: IN queries
Good thing I didn't rewrite my code before asking :-). I see the asynctools Jason posted, but I would be looking for a join (in the thread sense) that allows continued processing once all the results are back. On Fri, Sep 4, 2009 at 8:44 AM, Nick Johnson (Google)nick.john...@google.com wrote: Hi Jeff, IN queries are split up into multiple basic queries entirely in user code, so there's no latency or other benefit to using them over doing multiple queries yourself - it's just for convenience. In future, though, the MultiQuery interface may be extended to do its queries in parallel. -Nick On Fri, Sep 4, 2009 at 4:30 PM, Jeff Enderwick jeff.enderw...@gmail.com wrote: I read somewhere that IN queries are processed serially by the datastore. GOOGers, what is a rough rule-of-thumb on the benefit of using IN? For example, if the base RT latency for anything with the datastore is Nms, then could guesstimate that using N for a list of 3 is not a huge latency win, but using IN for 10 is. Does using IN substantially cut down on the api_cpu_ms used vs sequential queries? -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] The number of errors generated by your application every second
In the context dashboard, what precisely does 'errors' mean? I see an error frequency in the dashboard that is much higher than what I see in the logs. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Retrieving Source from GAE
dropbox is a nice/convenient option as well. GOOGers, this is a missing feature that many customers have assumed/expected, to their detriment. Personally, I have on off-site backups, but it seems that many aren't so fortunate (or are a little too nonki). Yeah, if the customer uses his/her head, this situation can be avoided. But the bottom line is that you are witnessing customer pain, and by not addressing it, you will inevitably loose customers. $.02 On Fri, Sep 4, 2009 at 5:43 PM, Tim Hoffmanzutes...@gmail.com wrote: On Sep 4, 12:48 am, ae gappengin...@gmail.com wrote: Hi, I had a couple of hard drives die one after the other in a shot period of time so I lost all my source code. Can someone from Google please PLEASE help me retrieve the source from GAE? Thanks. Unfortunatley if you don't install one of the various tools to retrieve code before you lose your code you are out of luck. If you don't want to use a public vcs I suggest you in the future tar / zip up you code and mail it to yourself in gmail. At least that way you will have an offsite backup. T --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: 30 Max simultaneous requests (maxThreads)?
There are application domains that spike severely. One example is AAA for corporate environments. There are large companies on the planet where *everyone* comes to work @ 8am +/-10min and logs in. Just one example from a prior life... On Wed, Sep 2, 2009 at 12:34 PM, Brandon N. Wirtzdrak...@digerat.com wrote: What would you need all those threads for in a web App? If you were crunching DNA, or calculating the path of particles from the big bang that would make sense... But if your App completes a request in less than 3 seconds, you would still be able to server 10 users a second, 600 users a minute, 864k requests a day. If having 3 threads sped this up to 1 second, the math would work out to the same number of request potentials. Do to the CPU time limits on a task, it is unlikely you would want 2000 threads because you'd burn through your Cycle limit just setting up those threads. I'm assuming from the Smiley after your comment about Cranking up the max threads, that you THINK you are doing something clever. In all likely unless you are doing some super computing apps on a CPU(s) with enough cores to support that number of threads you are just wasting cycles in thread management, and time slicing. -Brandon Wirtz -Original Message- From: google-appengine@googlegroups.com [mailto:google-appeng...@googlegroups.com] On Behalf Of Adligo Sent: Tuesday, September 01, 2009 5:34 PM To: Google App Engine Subject: [google-appengine] 30 Max simultaneous requests (maxThreads)? Hi, I am developing a app (or more than one) that I would like to host on Google App Engine, however the architecture of the app involves cranking up the maxThreads (I am using my home grown hosting which now has Tomcat set to 2,000 maxThreads :) ). For example (1 six+ year old machine in my basement) http://zeuhl.adligo.com/gwt_util_demo_v3_1/GwtDemo.html?show_log=true --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: high variance in cpu_ms for same GET URL data, constant api_cpu_ms
thanks - I expected that the api calls would use parallel processing, but the app/servelet itself is a single thread of execution. if I have api_cpu_ms of 74, and cpu_ms of 1500, then that gives 1426ms for the non-api (app/servelet usage), yes? I'm trying to grok how that would happen in a single thread in 965ms of wall-clock time. Jeff On Mon, Aug 31, 2009 at 11:30 AM, Jeff S (Google)j...@google.com wrote: Hi Jeff On Fri, Aug 28, 2009 at 1:08 AM, Jeff Enderwick jeff.enderw...@gmail.com wrote: Trolling my logs, I'm coming across cases where there is extreme (~10x) variance in cpu_ms for the exact same code flow, same GET URL and same data (not even any intervening writes to the datastore). I am logging my db.* function accesses, and I have factored out memcache too. For example: 92ms, 142cpu_ms, 74api_cpu_ms, followed by: 965ms, 1500cpu_ms, 74api_cpu_ms Q1: what could cause such a whopping delta? I am using Django, so perhaps template compilation? I used cprofile on the SDK with a similar data/result set, and the first page served was maybe ~2x subsequent pages in total time. Thoughts? Your idea of template compilation is along the same lines as my thinking. I can't say difinitively for this case but I would guess that you might be seeing a more expensive first request when a new instance of you app is being spun up. Q2: I am assuming the 1st number after the '200' is the wall-clock time-to-server. As the app is single threaded ... how is it able to burn 1500ms less 74ms in only 965ms? Most API calls make calls to distributed services which parallelize work across multiple machines, so it often easy to use more CPU time than wall clock time. If you want to see where the CPU usage is coming from, you can get information about CPU quota levels at any point within your code as documented here: http://code.google.com/appengine/docs/quotas.html#Monitoring_CPU_Usage_in_a_Request Thank you, Jeff Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: high variance in cpu_ms for same GET URL data, constant api_cpu_ms
Thanks Jeff - the app is 'el-cajon', version 4. Python runtime. On Mon, Aug 31, 2009 at 3:16 PM, Jeff S (Google)j...@google.com wrote: Hi Jeff, Ah I see, thanks for the details. I'm looking into this, would you mind sharing which runtime you are using, and the app ID? Cheers, Jeff On Aug 31, 1:52 pm, Jeff Enderwick jeff.enderw...@gmail.com wrote: thanks - I expected that the api calls would use parallel processing, but the app/servelet itself is a single thread of execution. if I have api_cpu_ms of 74, and cpu_ms of 1500, then that gives 1426ms for the non-api (app/servelet usage), yes? I'm trying to grok how that would happen in a single thread in 965ms of wall-clock time. Jeff On Mon, Aug 31, 2009 at 11:30 AM, Jeff S (Google)j...@google.com wrote: Hi Jeff On Fri, Aug 28, 2009 at 1:08 AM, Jeff Enderwick jeff.enderw...@gmail.com wrote: Trolling my logs, I'm coming across cases where there is extreme (~10x) variance in cpu_ms for the exact same code flow, same GET URL and same data (not even any intervening writes to the datastore). I am logging my db.* function accesses, and I have factored out memcache too. For example: 92ms, 142cpu_ms, 74api_cpu_ms, followed by: 965ms, 1500cpu_ms, 74api_cpu_ms Q1: what could cause such a whopping delta? I am using Django, so perhaps template compilation? I used cprofile on the SDK with a similar data/result set, and the first page served was maybe ~2x subsequent pages in total time. Thoughts? Your idea of template compilation is along the same lines as my thinking. I can't say difinitively for this case but I would guess that you might be seeing a more expensive first request when a new instance of you app is being spun up. Q2: I am assuming the 1st number after the '200' is the wall-clock time-to-server. As the app is single threaded ... how is it able to burn 1500ms less 74ms in only 965ms? Most API calls make calls to distributed services which parallelize work across multiple machines, so it often easy to use more CPU time than wall clock time. If you want to see where the CPU usage is coming from, you can get information about CPU quota levels at any point within your code as documented here: http://code.google.com/appengine/docs/quotas.html#Monitoring_CPU_Usag... Thank you, Jeff Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: high variance in cpu_ms for same GET URL data, constant api_cpu_ms
Okay - so ms != milliseconds, really (real-cpu-GHz * milliseconds)/1.2GHz. That clears it up. On Mon, Aug 31, 2009 at 4:27 PM, Jeff S (Google)j...@google.com wrote: Hello again Jeff, The CPU measurements for runtime are actually in cycles which would be performed by a 1.2 GHz processor in a second. We used to call these amounts megacycles, as described in our documentation: http://code.google.com/appengine/docs/quotas.html#Resources CPU time is reported in seconds, which is equivalent to the number of CPU cycles that can be performed by a 1.2 GHz Intel x86 processor in that amount of time. The actual number of CPU cycles spent varies greatly depending on conditions internal to App Engine, so this number is adjusted for reporting purposes using this processor as a reference measurement. For the runtime portion of your application (non API usage), the CPU is running faster than 1.2 GHz so it is possible to use more normalized CPU seconds (or megacycles if you prefer) in the amount of real time. This does seem a bit confusing though, so I'm open to ideas on how to make this more clear. Thank you, Jeff On Aug 31, 3:20 pm, Jeff S (Google) j...@google.com wrote: Nevermind, I forgot that you had mentioned Django :-) The app ID would still be helpful as well as example code if you feel comfortable sharing. Thank you, Jeff On Aug 31, 3:16 pm, Jeff S (Google) j...@google.com wrote: Hi Jeff, Ah I see, thanks for the details. I'm looking into this, would you mind sharing which runtime you are using, and the app ID? Cheers, Jeff On Aug 31, 1:52 pm, Jeff Enderwick jeff.enderw...@gmail.com wrote: thanks - I expected that the api calls would use parallel processing, but the app/servelet itself is a single thread of execution. if I have api_cpu_ms of 74, and cpu_ms of 1500, then that gives 1426ms for the non-api (app/servelet usage), yes? I'm trying to grok how that would happen in a single thread in 965ms of wall-clock time. Jeff On Mon, Aug 31, 2009 at 11:30 AM, Jeff S (Google)j...@google.com wrote: Hi Jeff On Fri, Aug 28, 2009 at 1:08 AM, Jeff Enderwick jeff.enderw...@gmail.com wrote: Trolling my logs, I'm coming across cases where there is extreme (~10x) variance in cpu_ms for the exact same code flow, same GET URL and same data (not even any intervening writes to the datastore). I am logging my db.* function accesses, and I have factored out memcache too. For example: 92ms, 142cpu_ms, 74api_cpu_ms, followed by: 965ms, 1500cpu_ms, 74api_cpu_ms Q1: what could cause such a whopping delta? I am using Django, so perhaps template compilation? I used cprofile on the SDK with a similar data/result set, and the first page served was maybe ~2x subsequent pages in total time. Thoughts? Your idea of template compilation is along the same lines as my thinking. I can't say difinitively for this case but I would guess that you might be seeing a more expensive first request when a new instance of you app is being spun up. Q2: I am assuming the 1st number after the '200' is the wall-clock time-to-server. As the app is single threaded ... how is it able to burn 1500ms less 74ms in only 965ms? Most API calls make calls to distributed services which parallelize work across multiple machines, so it often easy to use more CPU time than wall clock time. If you want to see where the CPU usage is coming from, you can get information about CPU quota levels at any point within your code as documented here: http://code.google.com/appengine/docs/quotas.html#Monitoring_CPU_Usag... Thank you, Jeff Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] high variance in cpu_ms for same GET URL data, constant api_cpu_ms
Trolling my logs, I'm coming across cases where there is extreme (~10x) variance in cpu_ms for the exact same code flow, same GET URL and same data (not even any intervening writes to the datastore). I am logging my db.* function accesses, and I have factored out memcache too. For example: 92ms, 142cpu_ms, 74api_cpu_ms, followed by: 965ms, 1500cpu_ms, 74api_cpu_ms Q1: what could cause such a whopping delta? I am using Django, so perhaps template compilation? I used cprofile on the SDK with a similar data/result set, and the first page served was maybe ~2x subsequent pages in total time. Thoughts? Q2: I am assuming the 1st number after the '200' is the wall-clock time-to-server. As the app is single threaded ... how is it able to burn 1500ms less 74ms in only 965ms? Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: synthentic keys - performance implications?
For posterity, one such gotcha is a case where Model instances work fine, but Expando instances can loose their additive attributes coming back out of the datastore. Switched to factory @staticmethod, and all is good now... On Tue, Aug 25, 2009 at 6:32 AM, Nick Johnson (Google)nick.john...@google.com wrote: On Mon, Aug 24, 2009 at 6:44 PM, Jeff Enderwick jeff.enderw...@gmail.com wrote: thanks - I got bit by those __init__ nuances over the weekend. I ended up passing an optional flag to the __init__ to say this is really a new() vs a datastore reconstitution. I del the optional flag from kwargs before calling the super __init__. In the datastore reconstitution case, I do nothing but call the super __init__. Does that cover the __init__ gotchas, or am I digging my own grave by not converting to a distinct create function? I don't know the gotchas well enough to say. I still think a factory function is your best bet. :) -Nick Johnson On Mon, Aug 24, 2009 at 3:56 AM, Nick Johnson (Google)nick.john...@google.com wrote: Hi Jeff, On Sat, Aug 22, 2009 at 7:24 PM, Jeff Enderwick jeff.enderw...@gmail.com wrote: Currently, one must put() in order to have obj.key() be valid. In some flows, I find my self having to put() object twice for this reason. If I make a synthetic key, it appears that I can avoid this: class Joker(db.Model): unused = db.StringProperty() def __init__(self): m = hashlib.sha1() m.update(str(time.time())) name = base64.b64encode(m.digest()) logging.debug(name=+name) db.Model.__init__(self, key_name=name) 1) GOOG folks - are there any performance downsides to taking this approach? Not really, no. 2) If no, are there any other environmental factors that might be fodder for the hash (user, etc)? I would recommend using uuid.uuid4().hex instead of a straight SHA1 sum. UUIDs are guaranteed to be unique. I would also recommend defining a class method called something like 'create' that generates the key name and calls __init__. There are subtle-use cases around __init__ and reconstructing entities from the datastore, and it's difficult to get right - much more straightforward to define a class method to construct new entities. -Nick Johnson Thanks, Jeff -- Nick Johnson, Developer Programs Engineer, App Engine -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: synthentic keys - performance implications?
thanks - I got bit by those __init__ nuances over the weekend. I ended up passing an optional flag to the __init__ to say this is really a new() vs a datastore reconstitution. I del the optional flag from kwargs before calling the super __init__. In the datastore reconstitution case, I do nothing but call the super __init__. Does that cover the __init__ gotchas, or am I digging my own grave by not converting to a distinct create function? On Mon, Aug 24, 2009 at 3:56 AM, Nick Johnson (Google)nick.john...@google.com wrote: Hi Jeff, On Sat, Aug 22, 2009 at 7:24 PM, Jeff Enderwick jeff.enderw...@gmail.com wrote: Currently, one must put() in order to have obj.key() be valid. In some flows, I find my self having to put() object twice for this reason. If I make a synthetic key, it appears that I can avoid this: class Joker(db.Model): unused = db.StringProperty() def __init__(self): m = hashlib.sha1() m.update(str(time.time())) name = base64.b64encode(m.digest()) logging.debug(name=+name) db.Model.__init__(self, key_name=name) 1) GOOG folks - are there any performance downsides to taking this approach? Not really, no. 2) If no, are there any other environmental factors that might be fodder for the hash (user, etc)? I would recommend using uuid.uuid4().hex instead of a straight SHA1 sum. UUIDs are guaranteed to be unique. I would also recommend defining a class method called something like 'create' that generates the key name and calls __init__. There are subtle-use cases around __init__ and reconstructing entities from the datastore, and it's difficult to get right - much more straightforward to define a class method to construct new entities. -Nick Johnson Thanks, Jeff -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Google App Engine and Dedicated IP
There is a related feature request, which you can star: http://code.google.com/p/googleappengine/issues/detail?id=792colspec=ID%20Type%20Status%20Priority%20Stars%20Owner%20Summary%20Log%20Component On Mon, Aug 24, 2009 at 1:13 AM, jackyaoye...@gmail.com wrote: Thanks, I'll try to port the site to GAE. On Aug 23, 11:05 pm, Barry Hunter barrybhun...@googlemail.com wrote: No this is not possible, for many reasons. A static IP would lose much of the advantage of AppEngine. You could if you want proxy the site though the server with the static IP, but seems kinda silly. On 23/08/2009, yaoye yaoye...@gmail.com wrote: Hi, guys, Would it be possible to bind a dedicated IP to my GAE-based site? I have a pr-4 site, and I wanna port it to GAE, but I'm worried about losing its ranking in Google for IP changing. All suggestions are welcome! Thanks a lot. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: https support
Star it, bro: http://code.google.com/p/googleappengine/issues/detail?id=792colspec=ID%20Type%20Status%20Priority%20Stars%20Owner%20Summary%20Log%20Component On Fri, Aug 21, 2009 at 10:03 PM, repairmanalau2...@gmail.com wrote: google guys, I don't know if you guys realize the implication of not getting a static IP? if I have a domain (abc.com) I cannot map it to the appspot subdomain without an IP and I have to resort to a redirect. And search engines are not friendly to redirects. And no serious website will ever want to redirect to blah.appsot.com it just looks unprofessional. So this lack of static IP is preventing google app engine from becoming actually useful like EC2 for real business. And that's why there is no serious apps or serious ecommerce systems running on your GAE so far despite of all the hype. It's almost like selling a car without headlights i.e. sounds like a small miss but it will prevent the car you are selling from being popular. no serious person can drive only during the day time. Please don't think of this static IP problem from the techie point of view, think from your 'customers' perspective. This is a show stopper! GAE is such a good concept and can revolutionize the industry but this lack of static IP is such a spoiler! On Aug 6, 7:27 am, Tony fatd...@gmail.com wrote: Fair enough. I'm fine with we're working on solutions, as long as it's not sorry, wait for the world to upgrade to FF3/IE8. On Aug 6, 10:21 am, Nick Johnson (Google) nick.john...@google.com wrote: Hi Tony, We're looking into solutions. It's not as simple as it may first appear - in the case of a service like Amazon EC2, your server resides at only a single physical location, and so does theIPaddress you rent. In contrast, App Engine apps are served from IPs at Google datacenters around the globe. -Nick Johnson On Thu, Aug 6, 2009 at 3:18 PM, Tonyfatd...@gmail.com wrote: Why not charge a monthly fee for apps to get astaticIP(like Amazon does)? Scarcity of supply seems like a bit of a cop-out to me - it's not apparent that a majority of app engine apps require this support. The customers you're losing because of this, however, are customers that plan to process e-commerce transactions online without looking like a phishing scam. Customers that make money on your service are more likely to spend money on your service. On Aug 5, 3:55 am, Nick Johnson (Google) nick.john...@google.com wrote: Hi J Singh, Due to the way SSL works, this is not currently possible without allocating anIPaddress for each App Engine domain that would use SSL - which itself isn't very practical due to IPv4 address scarcity. The latest version of SSL supports using a singleIPaddress for multiple sites with different certificates, but browser support for this version is not yet nearly widespread enough to make it a practical alternative. -Nick Johnson On Wed, Aug 5, 2009 at 5:39 AM, J Singhj.si...@earlystageit.com wrote: For an appengine-based site,https://abc.appspot.comiscurrentlysupported buthttps://www.abc.comcannotbe supported. I know there is a technical hurdle to cross but didn't know if any techniques had been proposed for being able to usewww.abc.comwithSSLconnections? Thanks. J Singh Managing Director Early Stage IT (978) 760-2055 http://www.earlystageit.com -- Nick Johnson, Developer Programs Engineer, App Engine -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] synthentic keys - performance implications?
Currently, one must put() in order to have obj.key() be valid. In some flows, I find my self having to put() object twice for this reason. If I make a synthetic key, it appears that I can avoid this: class Joker(db.Model): unused = db.StringProperty() def __init__(self): m = hashlib.sha1() m.update(str(time.time())) name = base64.b64encode(m.digest()) logging.debug(name=+name) db.Model.__init__(self, key_name=name) 1) GOOG folks - are there any performance downsides to taking this approach? 2) If no, are there any other environmental factors that might be fodder for the hash (user, etc)? Thanks, Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: ListProperty vs StringListProperty
So, if I want StringListProperty, but I don't want/need the indexing, then I ought use ListProperty(db.Text)? Thanks again, Jeff On Mon, Aug 10, 2009 at 11:35 AM, Nick Johnson (Google)nick.john...@google.com wrote: On Mon, Aug 10, 2009 at 7:25 PM, Jeff Enderwickjeff.enderw...@gmail.com wrote: Thanks Nick. So in the case below, will the datastore then maintain an index on alist by default? Yes - str and unicode instances get indexed by default. -Nick On Mon, Aug 10, 2009 at 3:35 AM, Nick Johnson (Google)nick.john...@google.com wrote: Hi Jeff, A StringListProperty is exactly equivalent, as far as the datastore is concerned, to multiple individual String properties. That is, an entity like this: MyEntity(astring=foo, alist=[foo, bar]) is stored in the datastore analagous to this: ((astring, foo), (alist, foo), (alist, bar)) -Nick Johnson On Sun, Aug 9, 2009 at 6:33 PM, Jeff Enderwickjeff.enderw...@gmail.com wrote: Hey all, is there any datastop/API overhead difference between these two? Is there any difference in index (or lack thereof) maintenance by the datastore? Thanks, Jeff -- Nick Johnson, Developer Programs Engineer, App Engine -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: ListProperty vs StringListProperty
Thanks again Nick. Sorry to be a PITA. We've been told that TextProperty is effectively the same as StringProperty w/out the indexing (although the online doc indicates some max length differences). What are the differences between ListProperty(db.Text) and StringListProperty(indexed=False), and what makes the latter better? On Tue, Aug 11, 2009 at 3:55 AM, Nick Johnson (Google)nick.john...@google.com wrote: On Tue, Aug 11, 2009 at 7:04 AM, Jeff Enderwickjeff.enderw...@gmail.com wrote: So, if I want StringListProperty, but I don't want/need the indexing, then I ought use ListProperty(db.Text)? A better option would be to use StringListProperty(indexed=False). -Nick Johnson Thanks again, Jeff On Mon, Aug 10, 2009 at 11:35 AM, Nick Johnson (Google)nick.john...@google.com wrote: On Mon, Aug 10, 2009 at 7:25 PM, Jeff Enderwickjeff.enderw...@gmail.com wrote: Thanks Nick. So in the case below, will the datastore then maintain an index on alist by default? Yes - str and unicode instances get indexed by default. -Nick On Mon, Aug 10, 2009 at 3:35 AM, Nick Johnson (Google)nick.john...@google.com wrote: Hi Jeff, A StringListProperty is exactly equivalent, as far as the datastore is concerned, to multiple individual String properties. That is, an entity like this: MyEntity(astring=foo, alist=[foo, bar]) is stored in the datastore analagous to this: ((astring, foo), (alist, foo), (alist, bar)) -Nick Johnson On Sun, Aug 9, 2009 at 6:33 PM, Jeff Enderwickjeff.enderw...@gmail.com wrote: Hey all, is there any datastop/API overhead difference between these two? Is there any difference in index (or lack thereof) maintenance by the datastore? Thanks, Jeff -- Nick Johnson, Developer Programs Engineer, App Engine -- Nick Johnson, Developer Programs Engineer, App Engine -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] api_cpu_ms (need to be sure I'm not being stupid)
long story short, I had a URL that was running 900/1500ms (app/api) CPU, so I had to dig into it. I was surprised by just how much CPU time some simple things are taking (or how much I think they're taking ;-). For example, with this class: class AssocObject(db.Model): urlPath = db.TextProperty() fileName = db.TextProperty() contentType = db.TextProperty() blob = db.BlobProperty() # Unused in query history -- copied from input. - kind: AssocObject ancestor: yes and the following snippet: fileObj = AssocObject(parent = node) db.put(fileObj) Just the single db.put() costs 250/210ms of CPU. Getting rid of the 'parent =' saves on 10s of ms. The total data set size is in the 10s of objects. I am not setting any of the variables in the class instance; the code really is create obj and store it at this point. Am I missing something? Is this level of CPU burn to be expected? Thanks, Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: ListProperty vs StringListProperty
Thanks Nick. So in the case below, will the datastore then maintain an index on alist by default? On Mon, Aug 10, 2009 at 3:35 AM, Nick Johnson (Google)nick.john...@google.com wrote: Hi Jeff, A StringListProperty is exactly equivalent, as far as the datastore is concerned, to multiple individual String properties. That is, an entity like this: MyEntity(astring=foo, alist=[foo, bar]) is stored in the datastore analagous to this: ((astring, foo), (alist, foo), (alist, bar)) -Nick Johnson On Sun, Aug 9, 2009 at 6:33 PM, Jeff Enderwickjeff.enderw...@gmail.com wrote: Hey all, is there any datastop/API overhead difference between these two? Is there any difference in index (or lack thereof) maintenance by the datastore? Thanks, Jeff -- Nick Johnson, Developer Programs Engineer, App Engine --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] ListProperty vs StringListProperty
Hey all, is there any datastop/API overhead difference between these two? Is there any difference in index (or lack thereof) maintenance by the datastore? Thanks, Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: simultaneous dynamic requests, max blob size, multi-blob?
Thanks Nick! On Wed, Jul 22, 2009 at 3:56 AM, Nick Johnson (Google)nick.john...@google.com wrote: Hi Jeff, When I said 10MB, I was referring to the 10MB request size limit, and pointing out that the forthcoming blob API will allow storing blobs larger than that size also. You can currently store anything up to 10MB by splitting the blob into multiple datastore entities, but it's impossible to accept an upload greater than 10MB at all. -Nick Johnson On Tue, Jul 21, 2009 at 9:26 AM, Jeff Enderwickjeff.enderw...@gmail.com wrote: Yeah, just trying to get clarification. Nick's email stated 10MB, Wikipedia says 1MB with a 10MB response limit. If Wikipedia is right and I can get 10MB in/out the door, I can always fragment (or wait, or if Nick says it is now 10MB ... :-). http://en.wikipedia.org/wiki/Google_App_Engine#Hard_limits On Mon, Jul 20, 2009 at 4:00 PM, Wooblegeoffsp...@gmail.com wrote: On Jul 18, 8:29 pm, Jeff Enderwick jeff.enderw...@gmail.com wrote: Is 10MB the current limit, or is it 1MB? I was under the impression that the max blob size was 1MB, and max HTTP response size was 10MB. Do I have it right? API calls are limited to 1MB unless that's been changed without an announcement fairly recently; you couldn't get more than 1MB into an entity even if the blob itself could hold it. -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: simultaneous dynamic requests, max blob size, multi-blob?
Yeah, just trying to get clarification. Nick's email stated 10MB, Wikipedia says 1MB with a 10MB response limit. If Wikipedia is right and I can get 10MB in/out the door, I can always fragment (or wait, or if Nick says it is now 10MB ... :-). http://en.wikipedia.org/wiki/Google_App_Engine#Hard_limits On Mon, Jul 20, 2009 at 4:00 PM, Wooblegeoffsp...@gmail.com wrote: On Jul 18, 8:29 pm, Jeff Enderwick jeff.enderw...@gmail.com wrote: Is 10MB the current limit, or is it 1MB? I was under the impression that the max blob size was 1MB, and max HTTP response size was 10MB. Do I have it right? API calls are limited to 1MB unless that's been changed without an announcement fairly recently; you couldn't get more than 1MB into an entity even if the blob itself could hold it. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] https support for dev_appserver.py?
Looks like dev_appserver.py doesn't support http (secure in app.yaml). Any thoughts about adding that? Or did I miss something? Thanks! Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Eating one's own dog food
There are those of us who are betting real $s that GAE is a real platform for serving an app that needs to scale. I am grateful that GOOG lets me host my glassblowing web site for free (scalability not an issue). I expect to pay money to GOOG for this platform if/when things go well. Hopefully I will pay them a lot :-). My $.02 is that the email-based QA support has been very good. The GAE Marketing department seems to need major surgery: - you have to set meet a feature roadmap (with dates), - you have to be able to provide reference customers. These are business basics. Jeff On Thu, Jul 16, 2009 at 7:38 PM, GenghisOnemdkach...@gmail.com wrote: Hi Juguang Your words provide a nice counter-balance to those who are not so happy with App Engine. Yes, you are correct in reminding us all that we should be grateful for something free. And yes App Engine has likely provided a whole bunch of young, inquiring minds with an amazing opportunity to tinker and learn on a global scale. But for some strange reason, I'm also reminded of that old saying...you get what you pay for. On Jul 16, 7:26 pm, Juguang XIAO jugu...@gmail.com wrote: Dogfooding, as wikipedia names it, is ideological, may not be practical. People, as well as company, lives practically for surviving first then chasing dreams. I have seen the positive movement from Google, did so much for developers. People may take it for granted, thinking the leader should do more. I am content with GAE, as it offers some free and exciting stuffs. I cannot ask Google to give more, unless he decided so. People can be happy when they are grateful. Technically speaking, I do not think Google offers its best to developers. It will be too costy to do so. Their mainstream businesses need to be maintained, and I guess each business unit has its own authority and freedom to do thing in their own way. Core businesses and technologies need to be protected. If you are not happy, go for Microsoft. ;-) I do not believe, people can run their serious business without serious pay. 6.5 hour per day CUP time is enough for casual applications. If you need more, try to create multiple account, schedule roster among them, sync your data among them. This is the solutions. If terms and conditions of use of GAE allows, there will be open souce projects for GAE clustering. Juguang On Fri, Jul 17, 2009 at 10:03 AM, GenghisOne mdkach...@gmail.com wrote: So it looks like there's an updated Google App Engine roadmap and guess what...no mention of full-text search. Doesn't that strike anyone as a bit odd? How can an emerging cloud computing platform not effectively address full-text search? And what's really odd is the absolute silence from Google...quite frankly, I don't get it. On Jul 16, 12:28 pm, Bryan bj97...@gmail.com wrote: This is a very interesting discussion. I would like to see some input from Google. On Jul 15, 10:20 am, richard emberson richard.ember...@gmail.com wrote: I understand that BigTable is behind GAE, but my concern is more with GAE performance and quotas. If GAE had existed when Larry and Sergey were developing their pagerack algorithm, would they have used GEA for evaluation? I have my doubts. They would quickly reach quota limits, way before they knew if they had a viable idea. Richard Tony wrote: Though I realize this is not exactly what you're asking, the concept of GAE is that it exposes some of the infrastructure that all Google applications rely on (i.e. Datastore) for others to use. So, in a sense, Google's various applications were using App Engine before App Engine existed. As far as I know, every Google service runs on the same homogeneous infrastructure, which is part of what makes it so reliable (and why the only available languages are Python and Java, languages used internally at Google). But I don't work there, so maybe I'm completely off-base. On Jul 15, 12:53 pm, richard emberson richard.ember...@gmail.com wrote: Eating one's own dog foodhttp:// en.wikipedia.org/wiki/Eating_one's_own_dog_food or in this case: Using one's own cloud. Amazon' cloud is based upon the IT technology they use within Amazon. Salesforce.com's Force.com offering is what they used to build their CRM system. These cloud vendors Eat their own dog food. If a cloud vendor does not use their cloud offering for their other products and/or internal systems, one would have to assume that the cloud is viewed as a technology ghetto within their own corporation - good enough for others but not for ourselves. So, concerning the Google App Engine, are other groups within Google clamoring to port or build their offerings on top of the App Engine? If so, please be specific, what Google products and infrastructure and what are the schedules
[google-appengine] Re: simultaneous dynamic requests, max blob size, multi-blob?
Is 10MB the current limit, or is it 1MB? I was under the impression that the max blob size was 1MB, and max HTTP response size was 10MB. Do I have it right? On Thu, Jul 16, 2009 at 2:15 AM, Nick Johnson (Google)nick.john...@google.com wrote: Hi Jeff, On Thu, Jul 16, 2009 at 3:33 AM, Jeff Enderwickjeff.enderw...@gmail.com wrote: 1) I saw that 30 was listed somewhere as the max simultaneous dynamic requests for an app. Is this really true? Even if I am paying, and I have a very popular app? I see the math indicating that one can retire 400/sec with 75ms latency, but I am sure that GOOG-internal apps aren't held to this limit of 30. Is this really a hard limit? It's not a hard limit, no. If your app scales large enough, this will be increased. 400 QPS is a very busy app, though. 2) Great to see that Service for storing and serving large files is on deck. Can y'all give any indication whether this will be accomplished by making blobs bigger (or adding some sort of uberblob)? Any indications on new max size would be great. It will definitely be capable of storing blobs larger than the current 10MB limit. For more details, you'll have to wait for the service to be released. -Nick Johnson 3) Along those lines, has anyone implemented something like multi-blob fragmentation/reassembly for storing serving things greater than 1MB? Thanks! Jeff -- Nick Johnson, App Engine Developer Programs Engineer Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] simultaneous dynamic requests, max blob size, multi-blob?
1) I saw that 30 was listed somewhere as the max simultaneous dynamic requests for an app. Is this really true? Even if I am paying, and I have a very popular app? I see the math indicating that one can retire 400/sec with 75ms latency, but I am sure that GOOG-internal apps aren't held to this limit of 30. Is this really a hard limit? 2) Great to see that Service for storing and serving large files is on deck. Can y'all give any indication whether this will be accomplished by making blobs bigger (or adding some sort of uberblob)? Any indications on new max size would be great. 3) Along those lines, has anyone implemented something like multi-blob fragmentation/reassembly for storing serving things greater than 1MB? Thanks! Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: why there is no way to build a key id?
Just star'd it -thx.
2009/6/24 Jeff S (Google) j...@google.com:
Hi Jeff,
This is an idea that we're aware of as well. If you could reserve the next
ID in advance, then you could actually do this in one put since multiple
entities could be sent in one batch :-) The workaround available now is to
use the key_name, but the difficulty becomes ensuring that the key_name is
unique. Here is a feature request which I'm aware of which is along these
lines (though the approach differs slightly):
http://code.google.com/p/googleappengine/issues/detail?id=1003
Thank you,
Jeff
2009/6/24 Jeff Enderwick jeff.enderw...@gmail.com
Hey Jeff - sorry for the confusion. The idea was that one would be
able to get the unique id from an GOOG and then do a db.put with that
id as an arg. For example, let's say I want to create two entities,
with each referring to each other. I need to do three db.put
operations:
a = Foo()
db.put(a)
b = Foo()
b.ref = a.key()
db.put(b)
a.ref = b.key()
db.put(a)
One would hope to be able to do this with two db.puts.
Thanks,
Jeff
2009/6/16 Jeff S (Google) j...@google.com:
The datastore does not allow key_names which begin with a digit in
order to avoid confusion with an ID, which is numerical. If you want
to use numeric key names, you could add a one letter prefix :-)
Happy coding,
Jeff
On Jun 16, 1:17 am, cryb cbuti...@gmail.com wrote:
Hi Jeff.
Thanks for your reply.. I really hope that in the near future
appengine will support setting key ids for entities.
You mentioned that I can use hooks in order to achieve my goal..
However I was more interested in a solution based on appengine java
sdk, and not on python hooks. Does appengine java sdk provide hooks or
some other similar mechanism?
It seems that for the moment I'll stick to generating key names.
One more question: I've tried to generate some entities with key names
on my local devappserver and I got a strange exception stating that I
can't create key names that start with a digit (?!?)... this holds on
google appengine production servers too or it's just a bug of
devappserver?
Thanks
On Jun 16, 2:45 am, Jeff S (Google) j...@google.com wrote:
Hi cryb,
As you noted, we do not currently allow the ID for a key to be set,
as we
ensure that the ID is unique for each existing entity. I recommend
using a
key name instead of an ID, as Antoniov suggeted, if possible.
It is technically possible to modify the key of an entity as it is
being
converted to a protocol buffer message before it is sent to the
datastore.
You could do this using hooks in the API proxy as described in this
article:http://code.google.com/appengine/articles/hooks.htmlAlsoit is
possible to
construct the key for the desired object if you know the ID in
advance.
class X(db.Model):
pass
# If you've already created the entity so you have the ID.
x_id = X().put().id()
# Instead of getting by ID, you can create the key manually.
k = db.Key.from_path('X', x_id)
Now you have the desired key without having fetched the object, but
the part
which the model class does not allow is setting the key yourself. So
you
could modify the protocol buffer message before it is written to the
datastore, but I don't recommend it.
The decision to allow setting key_names but not IDs is something we
may
revisit.
Happy coding,
Jeff
2009/6/12 cryb cbuti...@gmail.com
Hi.. that is to build key names... What I asked was why I can't
build
a key ID..
On Jun 12, 5:35 am, Antoniov nio@gmail.com wrote:
Use the code:
s = Story(key_name=xzy123)
Then you create an entity with the key name xzy123.
Check this:
http://code.google.com/intl/en-US/appengine/docs/python/datastore/key...
On 6月12日, 上午1时28分, cryb cbuti...@gmail.com wrote:
Does anyone know why it is possible to build a key name but NOT
a key
id? I know key IDs are used as autoincrements, but why can't I
just
override this mechanism and build my own key id?
Suppose I want to overwrite an existent entry in my table that
has a
key id I know, and also I want to keep that key id after
update...
because I can't just build a key id, I am forced to fetch that
entity,
modify it and write it back, instead of just write the updated
entity
with the key id I already know - so an additional read to the
datastore.
Is there an obscure reason for that? (both key names and key
ids are
prefixed with appid/kind as far as I know so there is no chance
of
collision with other apps/kinds)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google
[google-appengine] Re: HIPAA requirements vs. AppEngine security guidelines
I say go hire a HIPAA consultant who can answer such questions authoritatively. I've been through FIPS before, and you would not believe the odd lawyeresque contrivances used to get certified. With HIPAA you are in the same realm, and so you should hire yourself the appropriate barrister. $.02, Jeff On Tue, Jul 7, 2009 at 7:11 PM, GenghisOnemdkach...@gmail.com wrote: Andy Thanks for the heads-up... The link to that paper is here and it makes for a good read... http://awsmedia.s3.amazonaws.com/AWS_HIPAA_Whitepaper_Final.pdf Unfortunately after I skimmed through it I felt a little unsettled about AppEngine's security model...probably just my limited understanding of what's under the hood, but nonetheless security is kinda important and maybe its time to start asking some plain questions. For instance, here's one thing the Amazon whitepaper had to say about auditing... In designing a HIPAA-compliant system, customers should put auditing capabilities in place to allow security analysts to drill down into detailed activity logs or reports to see who had access, IP address entry, what data was accessed, etc. This data should be tracked, logged, and stored in a central location for extended periods of time in case of an audit. So can AppEngine enable this and if so how? My gut is telling me yes but there's still a nagging concern...How do I know if someone inside Google looked at my customers data? Is there some kind of *deep* logging mechanism of sorts? Thx much. BTW -- If Google has a comparable whitepaper, I'd very much appreciate the link. Thx much. On Jul 7, 4:01 am, Andrew Badera and...@badera.us wrote: There's a whitepaper by Amazon on the topic. Google it, it's been a few months since I looked at it, don't have a link offhand, sorry. Thanks- - Andy Badera - and...@badera.us - Google me:http://www.google.com/search?q=andrew+badera - This email is: [ ] bloggable [x] ask first [ ] private On Mon, Jul 6, 2009 at 5:17 PM, GenghisOnemdkach...@gmail.com wrote: Does anyone know if Amazon's EC2 platform is HIPAA-compliant? On Jul 6, 12:44 pm, richard emberson richard.ember...@gmail.com wrote: Not going to happen. The IT requirements for Google would cost far more than the couple of applications that might need HIPAA. They would have to have a completely separate group with their own machines, passwords, procedures, etc. with a real wall (both material wall and software/hardware wall) between the group and the rest of Google or all of Google would have to be HIPAA compliant. So, how much is it worth for Google? Not much. RME Ken wrote: Hi, I'm researching the feasibility of running a healthcare app on the AppEngine cloud. I've read through the AE terms of service and they don't say much about the actual security guidelines other than deferring to the boilerplate Google security policy. I have no doubt there are internal documents detailing the exact security guarantees provided by Google's infrastructure, but that information is not readily available to the public. It's been a full year since the last time HIPAA was discussed in this group. Now that SSL support has been enabled, data transfer constraints can be met with ease. So, what's the story today with GAE and HIPAA compliance? Are the App Engine's data storage and transfer mechanisms compatible with the guidelines set out by HIPAA? Google Apps documentation has quite a bit more security information, such as specifying annual SAS 70 Type II audits. I'm not familiar with this particular security audit, but some quick research seems to indicate that SAS 70 audit controls are mostly a superset of HIPAA guidelines. However, there are some aspects of HIPAA compliance that seem to be difficult to implement in a distributed database system, so any reassurances from the Google App Engine folks in this regard would be most appreciated. Thanks! Ken -- Quis custodiet ipsos custodes --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: why there is no way to build a key id?
Hey Jeff - sorry for the confusion. The idea was that one would be
able to get the unique id from an GOOG and then do a db.put with that
id as an arg. For example, let's say I want to create two entities,
with each referring to each other. I need to do three db.put
operations:
a = Foo()
db.put(a)
b = Foo()
b.ref = a.key()
db.put(b)
a.ref = b.key()
db.put(a)
One would hope to be able to do this with two db.puts.
Thanks,
Jeff
2009/6/16 Jeff S (Google) j...@google.com:
The datastore does not allow key_names which begin with a digit in
order to avoid confusion with an ID, which is numerical. If you want
to use numeric key names, you could add a one letter prefix :-)
Happy coding,
Jeff
On Jun 16, 1:17 am, cryb cbuti...@gmail.com wrote:
Hi Jeff.
Thanks for your reply.. I really hope that in the near future
appengine will support setting key ids for entities.
You mentioned that I can use hooks in order to achieve my goal..
However I was more interested in a solution based on appengine java
sdk, and not on python hooks. Does appengine java sdk provide hooks or
some other similar mechanism?
It seems that for the moment I'll stick to generating key names.
One more question: I've tried to generate some entities with key names
on my local devappserver and I got a strange exception stating that I
can't create key names that start with a digit (?!?)... this holds on
google appengine production servers too or it's just a bug of
devappserver?
Thanks
On Jun 16, 2:45 am, Jeff S (Google) j...@google.com wrote:
Hi cryb,
As you noted, we do not currently allow the ID for a key to be set, as we
ensure that the ID is unique for each existing entity. I recommend using a
key name instead of an ID, as Antoniov suggeted, if possible.
It is technically possible to modify the key of an entity as it is being
converted to a protocol buffer message before it is sent to the datastore.
You could do this using hooks in the API proxy as described in this
article:http://code.google.com/appengine/articles/hooks.htmlAlsoit is
possible to
construct the key for the desired object if you know the ID in advance.
class X(db.Model):
pass
# If you've already created the entity so you have the ID.
x_id = X().put().id()
# Instead of getting by ID, you can create the key manually.
k = db.Key.from_path('X', x_id)
Now you have the desired key without having fetched the object, but the
part
which the model class does not allow is setting the key yourself. So you
could modify the protocol buffer message before it is written to the
datastore, but I don't recommend it.
The decision to allow setting key_names but not IDs is something we may
revisit.
Happy coding,
Jeff
2009/6/12 cryb cbuti...@gmail.com
Hi.. that is to build key names... What I asked was why I can't build
a key ID..
On Jun 12, 5:35 am, Antoniov nio@gmail.com wrote:
Use the code:
s = Story(key_name=xzy123)
Then you create an entity with the key name xzy123.
Check this:
http://code.google.com/intl/en-US/appengine/docs/python/datastore/key...
On 6月12日, 上午1时28分, cryb cbuti...@gmail.com wrote:
Does anyone know why it is possible to build a key name but NOT a key
id? I know key IDs are used as autoincrements, but why can't I just
override this mechanism and build my own key id?
Suppose I want to overwrite an existent entry in my table that has a
key id I know, and also I want to keep that key id after update...
because I can't just build a key id, I am forced to fetch that
entity,
modify it and write it back, instead of just write the updated entity
with the key id I already know - so an additional read to the
datastore.
Is there an obscure reason for that? (both key names and key ids are
prefixed with appid/kind as far as I know so there is no chance of
collision with other apps/kinds)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: why there is no way to build a key id?
I second that. I have cases where I'm building a structure of multiple
objects, and I have to do a put() just to get the id, and then I need
to do another put() to the same object later on. Eliminating the 2nd
put() of the same object in the flow would also allow me to wrap it in
a transaction.
2009/6/16 Andy Freeman ana...@earthlink.net:
The decision to allow setting key_names but not IDs is something we may
revisit.
I hope that you're also considering some way to request and allocate
an unused id for a given path prefix. (That way we can get unique key
ids to specify.)
On Jun 15, 4:45 pm, Jeff S (Google) j...@google.com wrote:
Hi cryb,
As you noted, we do not currently allow the ID for a key to be set, as we
ensure that the ID is unique for each existing entity. I recommend using a
key name instead of an ID, as Antoniov suggeted, if possible.
It is technically possible to modify the key of an entity as it is being
converted to a protocol buffer message before it is sent to the datastore.
You could do this using hooks in the API proxy as described in this
article:http://code.google.com/appengine/articles/hooks.htmlAlso it is
possible to
construct the key for the desired object if you know the ID in advance.
class X(db.Model):
pass
# If you've already created the entity so you have the ID.
x_id = X().put().id()
# Instead of getting by ID, you can create the key manually.
k = db.Key.from_path('X', x_id)
Now you have the desired key without having fetched the object, but the part
which the model class does not allow is setting the key yourself. So you
could modify the protocol buffer message before it is written to the
datastore, but I don't recommend it.
The decision to allow setting key_names but not IDs is something we may
revisit.
Happy coding,
Jeff
2009/6/12 cryb cbuti...@gmail.com
Hi.. that is to build key names... What I asked was why I can't build
a key ID..
On Jun 12, 5:35 am, Antoniov nio@gmail.com wrote:
Use the code:
s = Story(key_name=xzy123)
Then you create an entity with the key name xzy123.
Check this:
http://code.google.com/intl/en-US/appengine/docs/python/datastore/key...
On 6月12日, 上午1时28分, cryb cbuti...@gmail.com wrote:
Does anyone know why it is possible to build a key name but NOT a key
id? I know key IDs are used as autoincrements, but why can't I just
override this mechanism and build my own key id?
Suppose I want to overwrite an existent entry in my table that has a
key id I know, and also I want to keep that key id after update...
because I can't just build a key id, I am forced to fetch that entity,
modify it and write it back, instead of just write the updated entity
with the key id I already know - so an additional read to the
datastore.
Is there an obscure reason for that? (both key names and key ids are
prefixed with appid/kind as far as I know so there is no chance of
collision with other apps/kinds)- Hide quoted text -
- Show quoted text -
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Google App Engine group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Deep entity groups and long key strings
Thanks Nick. If I go down the path of building ID-based URLs in the manner you suggest, will I likely have substantially shorter URLs than I would with str(node.key())? Thanks, Jeff On Mon, Jun 15, 2009 at 4:44 AM, Nick Johnson (Google)nick.john...@google.com wrote: Hi Jeff, You're correct that the only way to access a datastore entity is to provide the full path to it - the key is the entity's primary (and only) key. If you know the entity type of each entity in the heirarchy, though, you can extract the id of the entity in question and all its parents, and represent the ID as a delimiter-separated list of IDs, though - 1/2/3/4 - and regenerate the key yourself. -Nick Johnson On Sun, Jun 14, 2009 at 5:51 PM, Jeff Enderwick jeff.enderw...@gmail.com wrote: I have possibly deep entity groups (think tree structure), but I want a URL where the user can go to a specific node, like: http://viewnode/node-id. I'd like to use db.get() to directly access the node rather than query based on node-id. If I build the URL based on str(node.key()), then the strings can get very big. Key.from_path() seems to want the entire path from the parent. Is it really the case that the only way to get direct access to an interior node in an entity group is provide the entire path to that node? Thanks, Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] forward reference in db.ReferenceProperty()
Apologies in advance if this is obvious to the experienced Python programmer. It is clear that 20+yrs of C has worn grooves in my brain: class ClassA(db.Expando): extref = db.ReferenceProperty(ClassB) ... class ClassB(db.Expando): extref = db.ReferenceProperty(ClassA) is what I want. Putting a: class ClassB(db.Expando): pass at the top seems to do the trick when playing w/Python, but GAE throws a KindError on assignment, so clearly this ain't right... Thanks, Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Deep entity groups and long key strings
I have possibly deep entity groups (think tree structure), but I want a URL where the user can go to a specific node, like: http://viewnode/node-id. I'd like to use db.get() to directly access the node rather than query based on node-id. If I build the URL based on str(node.key()), then the strings can get very big. Key.from_path() seems to want the entire path from the parent. Is it really the case that the only way to get direct access to an interior node in an entity group is provide the entire path to that node? Thanks, Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] looking for a GOOG logo I can use on my site
Something similar to what Facebook offers (the connect button). I see the folks at typepad are using this: http://static.typepad.com/.shared/images/profile/openid_buttons/google.gif Does GOOG get mad if I use it? If I shrink it? Does GOOG offer images that have been okayed by the branding folks? Jeff --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
