[google-appengine] Re: a fundamental increase in App Engine transparency and locality is needed
Jon, I think we'd all agree that an automatically published report is better than a), or b). However, because there's Secret Sauce running behind App Engine, surely a myriad of infrastructure details that you'll never be allowed to publish detailed data on, c) will also probably never be enough. Or, well, I'll ask the question: what infrastructure was involved in the most recent maintenance, and what about its performance characteristics changed? Measuring/publishing various reports that we all *could* generate ourselves *is* helpful, but what of the "behind the scenes" stuff that can break? Based on increases in latency, I could come up with all kinds of explanations (I've worked with Hadoop for a number of years, so I know of the general types of odd conditions that come up with a cluster of machines attempting to work in concert but managed by failure-tolerant software). Did a network switch fail that caused a large fraction of the DataStore nodes to become unreachable? If so, would you ever be willing to publish switch performance analytics as part of the App Engine health status? Or, more generally, was it something misconfigured or that just doesn't perform as well as predicted? If misconfigured, can it be hot-updated, or is another downtime necessary to repair the mistake? The discussion of dedicated clusters does bring up a related question: Google apparently does have some cluster update process, as evidenced by the way features roll out to different users of Gmail at different rates. I assume, based on everything we've been (not) told about AppEngine, that AppEngine is essentially running as one single instance of whatever granularity Gmail and similar apps are maintained at Google. Could it be split into several? Then, presumably, maintenance and upgrade periods could be scheduled somewhat independently for different pools. Ok, enough conjecture on my part. My point is, I think a PR-sanitized message is going to still be necessary unless you guys truly disclose the inner workings of the infrastructure that provides AppEngine, which seems unlikely to ever completely happen. -Bryan On Mar 6, 2:37 pm, Jon McAlister wrote: > This is a very interesting thread. And yes, indeed, cluster stability > and uptime and transparency are all of the utmost importance to us > here on the App Engine team. > > I also agree that a fundamental increase in transparency would serve > everyone's interests. > > As far as I can tell, transparency through a crisis occurs in one of > three ways: (a) direct communication, (b) broadcasts of pr-approved > messages, and (c) raw data. > > As an engineer, I like (c) more than (a) or (b). It's objective, and > it can be distributed more efficiently. It's more efficient than (b) > because we don't have to wordsmith it or get it reviewed and approved, > it just goes out immediately. It's more efficient than (a) because it > happens automatically; you don't take up the time of an engineer who > would otherwise be busy trying to solve the problem. > > This is why our team has made it a priority to be very transparent > about real time and detailed performance data with our Status > Site:http://code.google.com/status/appengine. I encourage everyone who > cares about their application to spend a bit of time learning that > site. Make sure to click through to the detailed graphs as well. > Further, if there is more data you would like us to export, then let > us know. It is in everyone's best interests for us to export more raw > data. > > It is my personal belief that by continuing to iterate and improve > upon our Status Site that we will achieve the fundamental increase in > transparency, as opposed to Daniel's proposal of dedicated hardware, > direct communication, and promises as to when problems will be fixed. > Although, Daniel does bring up the excellent point that there is not a > particularly efficient way for developers to report live production > problems other than this group. Any thoughts on systems that we should > look into here to make this more efficient? > > Jon > > > > On Fri, Mar 6, 2009 at 9:25 AM, johnP wrote: > > > My experience has shown that customers are generally OK with a > > response: There is a problem. We are aware of it and people are > > working on it. It is estimated to be resolved in X time. > > > So if we (as vendors) are able to act as a positive PR army for > > AppEngine, we need to be quickly armed with the data necessary to > > accurately answer the above questions. > > > If vendors (on the front lines of PR Triage) don't have this data, a > > blame-game will start. No, we have no %$#$% idea what's wrong. No - > > those #$$#&^%$ are not saying anything. Etc. > > > So far, I'd say the following. Google has been almost good about > > keeping the information flowing. There have been periods of 'radio > > silence'; and the official status report usually reports that "There > > has been an anomaly but we have determined that nobody
[google-appengine] Re: How to read URL path values as parameter
You have (at least) 2 options:
1) Use the regexp in webapp to extract them, ie, your URL mapping:
("/process/([^/]+)/([^/]+)",MyProcessHandler),
...
Then MyProcessHandler.get() will get called with two extra arguments,
one set to each of the groups in the regexp
2) Parse path_info yourself. It's handed in as part of the webapp at
self.request.path_info
On Mar 6, 10:04 am, kshah wrote:
> I wanted to usehttp://xyz.appspot.com/process/chicago/il
> URL instead ofhttp://xyz.appspot.com/process?city=chicago&state=il
>
> How do I convert these path values as GET parameters?
>
> Thanks!
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: GQL Query involving key string
You don't actually need to use GQL, just use:
db.get('mykeystring')
On Feb 26, 9:21 pm, Audrey M Roy wrote:
> How do I write a GQL query that's something like this...
>
> SELECT * FROM Pet WHERE __key__='mykeystring'
>
> the above returns the error BadFilterError: invalid filter: __key__
> filter value must be a Key; received mykeystring (a unicode).
>
> Audrey
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Counting more than 1000 results.
There is apparently a bug in how the Cookbook handles < characters. I rewrote the recipe to only use > characters, but, I consider that to be a problem for posting code. In any case, the recipe should be updated. Hopefully, it will be useful to some. On Feb 24, 5:35 pm, djidjadji wrote: > Hi Bryan, > > http://appengine-cookbook.appspot.com/recipe/infinite-item-fetch-gene... > > Something is wrong with the layout of the recipe. > Some code lines are missing. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Counting more than 1000 results.
But, be mindful of the possibility of using the new remote_api to create such updated counters, rather than "cron", if it's a one-time change. With a few extra lines of Python and a machine with enough time to run the updates, out-of-date data structures can be brought up to date without having to perform the hijinks of incremental updates on the live site. For instance, with my deepFetch routine (see the Cookbook entry here: http://is.gd/kChr), it's just matter of writing the usual "for x in deepFetch(queryFun):" routine to iterate through and update all data structures. -Bryan On Feb 23, 7:54 pm, "Sebastian E. Ovide" wrote: > That's right. That is the only way... so you need to design the application > with counters in mind. > > If you need to add a counter that you have not planned, then you need to > create a "cron like" task that counts those records and store the > information in your new counters shards. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Global Time Synchronisation Guarantees
If you really just care about consistency, and you're actually thinking of using external requests (which, because of propagation delays would still not provide a very strong guarantee), why not just bite the bullet and use a single object to synchronize transaction? This will force you to write your code to avoid updating that object as aggressively as possible, but if you insist on a single accurate anything in your system, you're basically going to run into the same set of problems. Of course, if you just need the *user* to have a consistently ordered transaction log, that would be one object per user, and really shouldn't suffer all that terrible contention (unless the user is somehow regularly generating multiple invoices per second). On Jan 2, 1:39 pm, James Ashley wrote: > On Jan 1, 12:17 pm, Andy Freeman wrote: > > > > For a number of possible reasons (including the consistency of a > > > running balance), an accounts engine might have a basic specification > > > requirement that invoices are generated in sequence with ascending > > > invoice numbers *and* ascending invoice time/date. > > > Except that consistency doesn't require both. Either one is > > sufficient and trying to have both merely puts you in the situation of > > a person with two clocks - you don't know which one is correct. > > > Moreover, it's easy enough to fake an ascending time/date. (Take the > > max of the system time and epsilon past the most recent entry.) > > > If time is actually an issue, you can't use system time unless it's > > guaranteed to be accurate, not just consistent. For that, you're > > probably better off consulting an accurate time source. > > This was pretty much exactly my thought. Put your own central server > on the web and use urlfetch() to get timestamp/invoice numbers. > > -- James --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Using Model.get_or_insert to create unique values for entities
Much simpler would be to always declare a parent when creating the
original object. Since the parent need not exist, it might be possible
to do this without even creating the parent (the Model docs say you
can declare a parent using a Key object - can you create a Key object
without creating a model to match it?).
Here's the scheme:
Pick a random key for the parent.
Instantiate a Key object
Declare it as parent when creating your original object.
Whenever you need to re-create the object, just do it in a
transaction.
What about uniqueness of the parent? Well, since you only care that
you've deleted/created the single entity, it doesn't even matter if
the parent is unique - just that it's unlikely that another such
transaction is happening at the same time. If you choose a long enough
random parent key, this devolves to the birthday attack basically,
just make it a long string, since it doesn't much matter anyway.
If you *do* have to create the parent object first, just create it,
then delete it once the desired child object exists. Costs a bit more
in Datastore time, but still solves the problem.
On Oct 27, 8:47 pm, yejun <[EMAIL PROTECTED]> wrote:
> The new object's key path will get longer and longer and eventually
> over the limit. Also you need to know all its previous keys to be able
> to retrieve it by key name.
>
> I think the best practice to change primal key is create an
> intermediate transaction log object which belongs to old entity group
> and point it to the possible new key. Then do the actual create then
> delete the intermediate transaction log object.
>
> On Oct 27, 7:30 am, Jon McAlister <[EMAIL PROTECTED]> wrote:
>
> > You are correct that the best way to get around key restrictions on
> > starting with digits is to add some string prefix. However, the
> > problem with this code is that you also need to take care when
> > modifying the uid for a user. There is no way to change a datastore
> > key. Instead, you would have to create a new entity (with the new uid)
> > and delete the old one (with the old uid). And, since you're not using
> > entity groups here it won't be possible to do that transactionally. To
> > get around this, you could make the new entity a child of the old
> > entity and that would work (it's perfectly to delete a parent entity).
>
> > On Oct 25, 7:30 pm, Alexis Bellido <[EMAIL PROTECTED]> wrote:
>
> > > Hello everybody, I needed to store some information about my users (I
> > > don't need to use Google accounts) and created a model called FbUser,
> > > each entity in the model is a fbuser and each fbuser has a unique user
> > > uid, a field I call 'uid'.
>
> > > A uid can't appear more than once in the datastore but it could
> > > eventually change for a user, so it's not inmutable. It can't be used
> > > directly as a key_name because it always starts with a number, in fact
> > > is always a ten digit number.
>
> > > I want to make sure that I only store unique uid's in the datastore so
> > > I thought about Model.get_or_insert:
>
> > >http://code.google.com/appengine/docs/datastore/modelclass.html#Model...
>
> > > and found this suggestion by Dado (thanks a lot for it):
>
> > >http://groups.google.com/group/google-appengine/browse_thread/thread/...
>
> > > I implemented it for my case like this:
>
> > > class FbUser(db.Model):
> > > """
> > > You can then call FbUser.get_or_insert_by_uid('foo') and get back an
> > > FbUser instance with that unique identifier (it gets created if it
> > > does not yet exists).Model.get_or_insert is automatically wrapped in
> > > a transaction
> > > """
>
> > > uid = db.StringProperty(required=True)
>
> > > @staticmethod
> > > def get_or_insert_by_uid(uid):
> > > # prefix with unique identifier to qualify and avoid beginning
> > > with digits, which datastore does not accept
> > > key_name = 'uid:'+uid
> > > return FbUser.get_or_insert(key_name, uid=uid)
>
> > > And then I can 'get or create' a fbuser with uid '123' using this:
>
> > > FbUser.get_or_insert_by_uid('123')
>
> > > I've tested and it works but I want to be sure if this is the right
> > > way of doing it. What do you think?
>
> > > Thanks!
>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Shutdown deployed application?
I don't think you can as a first-order operation. But, you could easily upload a version of your app that is "shut down" (possibly with nice, simple, static messages to that effect). If you upload it under a different version string (change "version:" in app.yaml), then you'd be able to "shut down" the app by switching the version in the Versions section of the dashboard. Of course, anyone who can guess your other version string could still access the old app, so this isn't as complete as an actual app-logic level change. On Oct 11, 1:00 am, GMan <[EMAIL PROTECTED]> wrote: > Hi, > > Is there a way to "shutdown" the application once it isdeployed? Is > this possible thru the Dashboard? If so, is it also possible to > restart it, or can you simply re-deploy it? > > Thanks, > GMan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Still having trouble with high-cpu warnings for thumbnails.
-So are textProperties more efficient than StringProperties because they're not indexed? You'd have to find the talk from Google IO to be sure. I believe it was the one about scalability, in the QA section. But yes, that is my understanding. As I understand it, every field that's not a TextProperty or a BlobProperty are implicitly indexed (this is how all = conditions are dealt with in queries). So, whenever you write such an object, it will take longer (because of the index updates). Another way of thinking about it, is that if you never need to query on a single value, make it a TextProperty or BlobProperty, if possible. -Wouldn't adding etag--while increasing efficiency if I have the same users loading the same image again and again--actually decrease efficiency for users who are opening up an thumbnail for the first time? In that situation, I'd have another column for etags in my datastore being requested w/ every query. Yes, you absolutely should generate the etag when you save the thumbnail, and save it in the model itself. Caching it separately is however still desirable as you can then avoid pulling the rest of the data into memory if it's not needed, or you can opt to not cache the rest of the data at all, instead only caching the etag, to be more cache friendly. A quick and easy hack for this is to generate the etag before creating the Thumbnail model instance - and use that etag as the named key. Then, you can do lookup and caching based on the etag alone, where that makes sense. Unless you have some specific meaning in your ID already, this should simplify the "how to deal with etags" question quite a bit. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
