[google-appengine] Re: Sudden GAE Behavior Change! Urgent HELP needed!
On May 4, 11:17 am, Joshua Smith wrote: > I've worked around the problem by moving the tzinfo behavior into a helper > object and delegating all the calls, but it's not as tidy as this original > solution, so I'd still appreciate some advice: > > 1) What did I do wrong? Is there some fundamental design patten in python > that says you have to do a certain thing when you subclass in order to expose > an interface? What is that thing I didn't do? In general the Model class is not designed for multiple inheritance. In this case you were multiply inheriting from a built-in type (tzinfo) and there are various CPython restrictions on m.i. from those; the error message you got came from the machinery related to that. What we did was actually make Model *more* correct around some other m.i. scenarios, but unfortunately this broke your code that until now had been luckily avoiding the landmines. The best solution (as you have found) is definitely to refactor your code and avoid m.i.; alternatives that happen to work (but run similar risks in the future) include swapping the inheritance order or adding a dummy __new__ to your class: def __new__(cls, *a, **k): return object.__new__(cls) However I cannot promise that this really works correctly nor that it will keep working in the future -- one of the downsides of using such a dynamic language as Python is that occasionally things happen to work that weren't intended to work... > 2) Why is this code OK in development, but not in production? The code > that's throwing the exception (that call to __new__ in __init__.py) doesn't > even exist in the development python code. We're working on the 1.5.0 release. As Marzia explained last release (http://groups.google.com/group/google-appengine-python/browse_thread/ thread/a3664fc8171ebaa8/c1228f138c8069d6) we can't push all components simultaneously so we typically upgrade the production runtime ahead of the SDK. (Do note that the 1.5.0 prerelease SDK does have this code. It is not perfect however. http://code.google.com/p/googleappengine/downloads/list) > 3) What's up with this changing during "system maintenance" and not as part > of an SDK update? I can't be the only one who's code unexpectedly stopped > working. It had nothing to do with the system maintenance; the 1.5.0 push to production was unrelated. Again, sorry for your problems and glad you found a fix already! --Guido van Rossum > -Joshua > > On May 4, 2011, at 10:10 AM, Joshua Smith wrote: > > > > > > > > > I have some code that has been working for a really long time, that > > suddenly started throwing an exception. > > > In my app, I have to keep some different time zones around, so I have a > > Model that carries the interesting stuff, and it acts as a tzinfo for > > convenience. > > > Like this: > > > class RegionModel(db.Model, datetime.tzinfo): > > name = db.StringProperty() > > gmtOffset = db.IntegerProperty() > > tzName = db.StringProperty() > > tzDstName = db.StringProperty() > > > def utcoffset(self, dt): > > return datetime.timedelta(hours=self.gmtOffset) + self.dst(dt) > > > etc. > > > This has been working fine, but starting at 5/3 at 15:50, I'm getting this > > exception: > > > 2011-05-03 15:50:27.934 > > datetime.tzinfo.__new__(RegionModel) is not safe, use object.__new__() > > Traceback (most recent call last): > > File > > "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__i > > nit__.py", line 698, in __call__ > > handler.get(*groups) > > File "/base/data/home/apps/towngovernment/37.349853758361595417/main.py", > > line 1691, in get > > default = board.anyTown().nextLikelyMeetingTime() > > File "/base/data/home/apps/towngovernment/37.349853758361595417/main.py", > > line 248, in nextLikelyMeetingTime > > t = self.nextLegalMeetingTime() > > File "/base/data/home/apps/towngovernment/37.349853758361595417/main.py", > > line 242, in nextLegalMeetingTime > > now = self.region.currentLocalTime() > > File > > "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init_ > > _.py", line 3491, in __get__ > > instance = get(reference_id) > > File > > "/base/python_runtime/python_lib/versions/1/google/appengine/ext/db/__init_ > > _.py", line 1471, in get > > return get_async(keys, **kwargs).get_result() > > File > > "/base/python_runtime/python_lib/versions/1/google/appengine/datastore/data > > store_rpc.py", line 644, in get_res
[google-appengine] Re: Sudden GAE Behavior Change! Urgent HELP needed!
On May 4, 12:12 pm, Joshua Smith wrote: > Fair enough. > > I'd recommend that you update the documentation of db.Model to point out that > "the Model class is not designed for multiple inheritance" > if it doesn't already say that somewhere I didn't see. Will get this taken care of by our doc team. > My workaround right now is to have my Model create a helper object and use > manual delegation patterns to get the calls done. It's kind of gross. Is > there a pythony way that would allow my model to present itself as a tzinfo, > without using multiple inheritance? I don't mind having the delegate inside > my model, but I'd rather not have to have all my callers get the delegate > through an extra call (or, in principle, even know about the delegate). I'm not fully following your use case, but perhaps it could be done the other way around? I.e. create a subclass of tzinfo that has a reference to a Model instance? --Guido van Rossum -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
On Jul 12, 11:16 pm, Max wrote: > Thanks Alfred, > > For Datastore Plus project, is there (or will there be) a Java version? No, NDB is Python specific in many ways. Java already has a decent async datastore API based on Futures: http://code.google.com/appengine/docs/java/datastore/async.html --Guido van Rossum (not Alfred, but Datastore Plus' author :-) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: What is the datastore charges for db.GqlQuery under new pricing scheme?
On Oct 25, 8:22 am, ramu wrote: > As I understand, query result are fetched in batches so even if we process > only couple of entities from a query, does it charge for the complete batch > operation ?? > > What is the datastore-read-count if i call query_object.get(), which returns > me just the 1st entity ? > > It is not exactly clear about Query calls on this > page.http://www.google.com/enterprise/cloud/appengine/pricing.html When you call get() on a query, the batch size is set to 1, so no worries. Similar if you do query_object.fetch(N) -- the batch size is adjusted to avoid fetching more than N objects. The only time where you would be fetching unneeded options is if you use a for-loop over the query and then break. E.g. the following is a bad way to fetch the first 3 hits: # Don't do this! hits = [] for entity in query_object: if len(hits) >= 3: break hits.append(entity) # Don't do this! --Guido van Rossum -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: [appengine-python] Re: 1.6.0 Prerelease SDKs are out
That's a known issue and it will be fixed in the final 1.6 SDK. --Guido On Wed, Nov 2, 2011 at 09:55, Kyle Finley wrote: > Sorry if this is a duplicate. > The development server seems to be ignoring content-type headers when using > the python27 runtime. > Posting application/json is read as application/x-www-form-urlencoded in > python27, but runtime: python reads it as application/json. > Is this a know issue or am doing something wrong in my setup? > Thanks, > Kyle > > -- > You received this message because you are subscribed to the Google Groups > "google-appengine-python" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine-python/-/GmmHKY-4gnwJ. > To post to this group, send email to > google-appengine-pyt...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-python+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-python?hl=en. > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: [appengine-python] Re: 1.6.0 Prerelease SDKs are out
On Tue, Nov 1, 2011 at 12:18, Дмитрий Лисовский wrote: > And what about <>? You can put it in your app.yaml, but the dev_appserver.py will not handle multiple requests in parallel, sorry. -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: [appengine-python] Re: 1.6.0 Prerelease SDKs are out
On Thu, Nov 10, 2011 at 15:54, Greg Coladonato wrote: > I have a nagging fear that this is a stupid question, but after reading > release notes and FAQs I still haven't found an answer. > > I just installed Python 2.7 on a new laptop. When I try to install the new > Python SDK, I get a warning that I need Python 2.5 installed in order to > run the SDK. Is that expected behavior? > > I ignored the warning and pressed on. Then, when I try to run the SDK, I > get a "python not found" type of error. > > I had assumed that 'support for Python 2.7' meant that Python 2.5 was no > longer needed by the SDK, is this a valid assumption? > Sorry, no, that's not what we meant when we announced Python 2.7 support. We meant that we now support both the Python 2.5 and Python 2.7 runtimes. The default remains 2.5; in order to use the 2.7 support you must put "runtime: python27" in your app.yaml file; and you still have to arrange to run the dev_appserver using Python 2.7, e.g. by using "python2.7 /dev_appserver.py". Separately, it sounds like you installed Python 2.7 in such a way that it is not found as "python" on the default path. --Guido Thanks everyone, > Greg > > -- > You received this message because you are subscribed to the Google Groups > "google-appengine-python" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine-python/-/2L8Vax_QRgYJ. > > To post to this group, send email to > google-appengine-pyt...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-python+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-python?hl=en. > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] NDB release 0.9.5 is out
I have released NDB 0.9.5. Please give it a try. I plan to include this version into the next App Engine SDK (1.6.2, due out later this month). About NDB: NDB is a new, experimental Python client library for the App Engine datastore. It is fully open source. I am looking for feedback and contributions from users, but please understand that it is not yet officially supported. A version of NDB is included in the official App Engine runtime and SDK with experimental status, but those always lag behind somewhat More Info: Tutorial: http://goo.gl/9Teuh Download: http://code.google.com/p/appengine-ndb-experiment/downloads/detail?name=appengine-ndb-experiment-0.9.5.zip Project page: http://code.google.com/p/appengine-ndb-experiment/ Issue tracker: http://code.google.com/p/appengine-ndb-experiment/issues/list Mailing list / group: https://groups.google.com/group/appengine-ndb-discuss?pli=1 Check out the repo: hg clone https://code.google.com/p/appengine-ndb-experiment/ Release notes: Various minor issues fixed over the holidays. - Issue 63: Add PickleProperty, JsonProperty. - Improve type check for ancestor parameter to query(). - Fix kind-less ancestor queries. - Issue 121: Do not die on *StructProperty equal to None. - Issue 119: Pre-compute the subclass when deserializing PolyModel. - Issue 123: Support custom property inside repeated StructuredProperty. - Issue 117: Simplify policy for updating cache with query results. - Issue 124: Don't log webob exceptions at alarming levels. - Issue 126: Improve repr() of Futures. - Issue 127: Fix Deadlock error in code that doesn't use @context.toplevel. -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: caching and in-transaction queries
Let me widen the scope of this question, since I'm not convinced that this is NDB-specific (though I'm also not convinced that it isn't). First, can you rerun the tests with NDB's caching turned off? INSIDE tx(), do ctx = tasklets.get_context() ctx.set_cache_policy(False) ctx.set_memcache_policy(False) (You also have to add from ndb import tasklets to the top.) Let me know if that changes the outcome, and how. If it does, it is an NDB issue; if it doesn't, it is not specific to NDB. If it isn't an NDB issue, there are all sorts of subtle semantics associated with transactions that I can't quite enumerate, and it could well be that ancestor queries inside transactions always query the state at the start of the transaction. --Guido On Tue, Jan 10, 2012 at 01:48, Amy wrote: > hi, > > I'm curious about something I'm seeing with in-transaction object > caching, and want to check that it's the intended behaviour. > > More detailed code is in this gist: > https://gist.github.com/8bc6bdeb8c4734b54bc9 > > Suppose we have two models, Game and Participant, where participant > entities are children of a game entity, and there is a method of Game, > participants(), to fetch all participants of a game via an ancestor > query on Participant: > > class Game(model.Model): > > def participants(self): > return Participant.query( > ancestor=self.key).fetch() > > If from within a transaction, I call game.participants(), modify the > returned participant objects, 'put' them, then -- still in the > transaction-- call game.participants() again, the second call does not > return the modified objects. > > On the other hand, if in the same transaction I fetch the participant > objects directly by key (after modification and a 'put'), the fetched > objects *do* reflect the modification. So presumably they are being > fetched from the cache by key. > The TestHandler.get() method in the gist (https://gist.github.com/ > 8bc6bdeb8c4734b54bc9) steps through these tests. > > This seems a bit inconsistent, as caching is being used in-transaction > in one case but not the other. However, this might be how it is > intended to work? I know that normally (without caching), queries > made in a transaction do access the pre-transaction state. > > --Amy > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: NDB query.filter
On Jan 10, 12:49 am, Moises Belchin wrote: > Hi all, > > I'm using NDB query filter and I want to know If it possible store > filters on list and then pass them to NDB.query.filter. > See code below: > > ## BEGIN CODE > ### > > from google.appengine.ext.ndb import query > from model import MyModel > > year_from = 2012 > month_from = 1 > day_from = 1 > > year_to = 2012 > month_to = 1 > day_to = 9 > > filterDate = [] > > for i in range(day_from, day_to): > filterDate.append(query.AND(MyModel.day == i, > MyModel.month == > month_from, > MyModel.year == > year_from)) > > q = MyModel.query() > q = q.filter(query.OR(filterDate)) # This raise: TypeError: Cannot > filter a non-Node argument; > result = q.fetch(20) > > ## END CODE > ### > > My questions: > 1. How I convert filterDate to Node argument if possible? > 2. Is this code correct and possible with NDB? > 3. If not, What is the best practise to implement something similar to > code above? You're almost there. All you need to do is to change the OR call as follows: q = q.filter(query.OR(*filterDate)) This is because OR() doesn't take a list of nodes, it takes one or more arguments each of which must be a node. The filterDate variable is a list of nodes. By using *filterDate you invoke Python's variable argument list feature which generates a separate argument for each item in the list. Good luck with your query! --Guido van Rossum -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] NDB 0.9.6 is released
I've released NDB 0.9.6. If all goes well this release will also be included in the 1.6.2 App Engine SDK. Download it at http://code.google.com/p/appengine-ndb-experiment/downloads/detail?name=appengine-ndb-experiment-0.9.6.zip Release notes: - Issue 141: Speed up fetch(), especially for keys_only=True. - Issue 140: Fix query for recursive structured property. - Issue 138: Fix for subproperty=None in repeated structured property. - Issue 125: Support prospective search API. - Issue 108: Remove get_or_insert() from Context. - Issue 117 (again): Improve cache/query interaction. - Explicit batch_size sets default for prefetch_size. - Add urlfetch support to Context. - Issue 111: Allow specifying a kind for KeyProperty. - Issue 129: FilterNode should convert new Key to old Key. - Add support for BlobKey in Expando (an old TODO). - Issue 132: Use either json or simplejson in JsonProperty. - Issue 131: Suppress debug messages about threading.local. __ (*) About NDB: NDB is a new, experimental Python client library for the App Engine datastore. It is fully open source. I am looking for feedback and contributions from users, but please understand that it is not yet officially supported. A version of NDB is included in the official App Engine runtime and SDK with experimental status, but those always lag behind somewhat. More Info: Tutorial: http://goo.gl/9Teuh Async tutorial: http://goo.gl/uFXEG Project page: http://code.google.com/p/appengine-ndb-experiment/ Issue tracker: http://code.google.com/p/appengine-ndb-experiment/issues/list Mailing list / group: https://groups.google.com/group/appengine-ndb-discuss?pli=1 Check out the repo: hg clone https://code.google.com/p/appengine-ndb-experiment/ -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.6.2 Pre-Release SDKs Out
On Jan 25, 9:39 am, someone1 wrote: > On a side note, will > google.appengine.ext.ndb.utils.DEBUG be set to True or False in > production? It will stay True. However feel free to set it to False in your app after you've debugged it (your app, that is :-). This will reduce the amount of work done to save debug info in every Future. PS. A better place to ask questions about NDB is appengine-ndb- discuss. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: [appengine-python] 2012 US PyCon -- GAE Related Sprints
On Mon, Feb 20, 2012 at 22:46, Robert Kluin wrote: > Hey Everyone, > Myself and several colleagues are going to be at pycon, and sticking > around for a few days of sprints after the main conference is over. > We're planning to discuss and work on several App Engine related > projects during that time. Some of the projects are pretty neat, and > should benefit the larger App Engine Python community. We'd also love > to have some help or more ideas for cool things to work on! If you're > going to be at PyCon and sticking around, feel free to sign up or just > catch one of us at the conference. Feel free to stop by even if you'd > just like to nerd-out about App Engine / Python! Hi Robert, Let's make sure we meet up in Santa Clara! I'll try to make it to some of the sprints though I may have to divide my attention between Python core and App Engine... > Some current items on the project list: > - An ext.db mock for "unit" tests that offers basic functionality > (including querying!) without the poor performance of the SDK stub. I hope you didn't literally mean ext.db -- please mock at a lower level so that NDB can also transparently benefit from this. (Note that the latest trend seems to be partial use of NDB, possibly prompted by the User model in webapp2_extras.) > - Work on PyAMF and RTMPy. > - Since we'll soon have pull-queue tagging, I want to get out the > updates I've been holding back for slagg. > - I'd like to put together a Cassandra stub for the SDK. I made > good progress on this last year, but it has fallen behind. > > > I've added our "spint" to the pycon sprints page, see "APP ENGINE > RELATED PROJECTS (COMMUNITY RUN)" > https://us.pycon.org/2012/community/sprints/projects/ > > > > Robert > > -- > You received this message because you are subscribed to the Google Groups > "google-appengine-python" group. > To post to this group, send email to google-appengine-pyt...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine-python+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine-python?hl=en. > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: [appengine-python] 2012 US PyCon -- GAE Related Sprints
On Feb 22, 9:51 pm, Robert Kluin wrote: > On Wed, Feb 22, 2012 at 12:18, Guido van Rossum wrote: > > On Mon, Feb 20, 2012 at 22:46, Robert Kluin wrote: > >> Some current items on the project list: > >> - An ext.db mock for "unit" tests that offers basic functionality > >> (including querying!) without the poor performance of the SDK stub. > > > I hope you didn't literally mean ext.db -- please mock at a lower > > level so that NDB can also transparently benefit from this. (Note that > > the latest trend seems to be partial use of NDB, possibly prompted by > > the User model in webapp2_extras.) > > Hey Guido, > Sounds great. Perhaps we can discuss how to best accomplish this a > little bit at pycon! I've got some basic ideas, but hopefully > together we can come up with a better strategy. Absolutely. Grab me when you see me. --Guido -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Official NDB docs now online! And other exciting NDB news
I am proud to announce that Google's amazing team of tech writers has published the first version of the official NDB documentation for App Engine: http://code.google.com/appengine/docs/python/ndb/ This means that I am no longer maintaining the provisional docs I wrote linked to from the NDB project page: http://code.google.com/p/appengine-ndb-experiment/ If you find anything unclear in the docs, please post your feedback to appengine-ndb-disc...@googlegroups.com or use the NDB issue tracker: http://code.google.com/p/appengine-ndb-experiment/issues/list Of course we're not done. There's a series of changes still on the way, see: http://code.google.com/p/appengine-ndb-experiment/source/browse/RELEASE_NOTES I also hope to publish a more complete guide to converting existing old db apps to NDB; for now, you can continue to use the cheat sheet for guidance on this topic: https://docs.google.com/document/d/1AefylbadN456_Z7BZOpZEXDq8cR8LYu7QgI7bt5V0Iw/edit And please look forward to an announcement that NDB is no longer experimental, though I can't promise when that will be. Until then, beware that NDB currently *is* experimental and subject to API changes that may break your app without warning. I'll try to minimize these and draw attention to the inevitable ones on these mailing lists. One issue (which will go live in the next SDK) that I'd like to call out involves pending changes to transaction interactions: https://groups.google.com/group/appengine-ndb-discuss/msg/7b5bb6c8bac1a6b5 -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] NDB 0.9.9 released
Leading up to the (still :-) hopeful elevation of NDB's status out of experimental to fully supported in the next App Engine SDK release, I am pleased to release NDB 0.9.9. Download: http://code.google.com/p/appengine-ndb -experiment/downloads/detail?name=appengine-ndb-experiment-0.9.<http://code.google.com/p/appengine-ndb-experiment/downloads/detail?name=appengine-ndb-experiment-0.9.8.zip> 9.zip<http://code.google.com/p/appengine-ndb-experiment/downloads/detail?name=appengine-ndb-experiment-0.9.8.zip> *INCOMPATIBILITIES:* * * *Be aware that the following incompatible changes will also be included in the next SDK and runtime.* - Issue 163<http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=163>: String/TextProperty should not allow writing non-UTF-8 text. - Issue 139<http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=139>: Enforce size limits for indexed text/blob properties. - Issue 156<http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=156>: Changed query cache policy (again). - Issue 162<http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=162>: Raise an error on attempting to create a nested transaction. - Issue 161<http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=161>: Make get_or_insert() use existing transaction if one exist. New feature: - Issue 158<http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=158>: Support GenericProperty(compressed=True). Other fixes: - Issue 164<http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=164>: Reduce log spam for exceptions in production. - Issue 159<http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=159>: Make Model.__init__() signature more robust. - Fixed index_list() tests (were broken due to a last-minute runtime change). *About NDB: * NDB is a new, experimental Python client library for the App Engine datastore. It is fully open source. I am looking for feedback and contributions from users, but please understand that it is not yet fully supported. A version of NDB is included in the official App Engine runtime and SDK with experimental status, but those always lag behind somewhat. (SDK 1.6.3 contains NDB 0.9.8.) *More Info: * Official docs (for 0.9.8): http://code.google.com/appengine/docs/python/ndb/ Tutorial (no longer maintained): http://goo.gl/6n6RO Async tutorial (no longer maintained): http://goo.gl/uFXEG Project page: http://code.google.com/p/appengine-ndb-experiment/ Issue tracker: http://code.google.com/p/appengine-ndb-experiment/issues/list Mailing list / group: https://groups.google.com/group/appengine-ndb -discuss?pli=1 Check out the repo: hg clone https://code.google.com/p/appengine-ndb -experiment/ -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: [appengine-python] Re: NDB 0.9.9 released
On Sat, Mar 3, 2012 at 15:39, Kaan Soral wrote: > "NDB is an experimental, innovative, and rapidly changing new feature for > App Engine. Unfortunately, being on the bleeding edge means that we may make > backwards-incompatible changes to NDB. We will inform the community when > this feature is no longer experimental." > > How much of this warning will still be valid? Good question. I wish I could guarantee that the 5 issues mentioned in my original email under "INCOMPATIBILITIES" are the last incompatible changes, but of course it's never done until it's done, and it's *possible* that we'll find some more things that we'd like to change before drawing the line in the sand. However you can tell from the list of incompatibilities that they are definitely edge cases. The changes in transaction semantics are the most serious IMO, but it's always possible that someone has accidentally put strings over 500 characters in a StringProperty (or over 500 bytes in an indexed BlobProperty). Still, no methods, functions or classes were renamed, and no signatures changed. I hope to continue this trend. In any case, I can't remind people enough that (at least) these incompatibilities will hit production when the 1.6.4 SDK goes out (actually production will see them a few days earlier) so please review your code, and if possible test with NDB 0.9.9. I also hope that 1.6.4 will see NDB's "experimental" label removed -- but no guarantees! -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Beware: slight NDB incompatibilities coming soon
(If you're using Java or if you are not yet using NDB with Python you can ignore this post.) Now that the 1.6.4 prerelease SDK is public (http://code.google.com/p/googleappengine/downloads/detail?name=google_appengine_1.6.4_prerelease.zip) I'd like to remind NDB users that the next version of NDB will contain a few incompatibilities. These are not in API signatures but in the behavior of certain operations. It is particularly important to test your app ahead of time with the prerelease SDK because, as usual, the production runtime will be switched to the SDK 1.6.4 behavior at some point later this week (possibly starting tomorrow!), well *before* the final SDK 1.6.4 is announced. (1) NESTED TRANSACTIONS In SDK 1.6.3, calling get_or_insert() or transaction() (or their async counterparts) starts a *new* *independent* transaction. This is usually not what you want -- two transactions are likely to conflict with each other if they use the same entity group. Therefore in SDK 1.6.4, get_or_insert() will join the current transaction if there is one, and transaction() will raise an exception. If there is no current transaction their behavior is unchanged from before. Note that code using the @ndb.transactional decorator is not affected, since this decorator already joins the current transaction if there is one. (2) STRICTER PROPERTY ENFORCEMENTS In SDK 1.6.3, some invalid property values were not rejected; in SDK 1.6.4 the following will be rejected when written: - StringProperty(): values longer than 500 characters - BlobProperty(indexed=True): values longer than 500 bytes - StringProperty() and TextProperty(): 8-bit string values that aren't proper UTF-8 (3) INTERACTION BETWEEN CONTEXT CACHE AND QUERY RESULTS In SDK 1.6.3, under some circumstances a query result could overwrite a value in the context cache. This was deemed undesirable (read http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=156) and in SDK 1.6.4, the context cache will have priority. (Queries do not interact with memcache.) (4) STRICTER TRANSACTION OPTIONS In SDK 1.6.3, it was possible to pass transaction-related options to non-transaction APIs (e.g. ent.put(retries=3)); these were silently ignored. In SDK 1.6.4 these will be rejected. Also, if you want to construct an Options object including transaction-related options, you must now use ndb.TransactionOptions(option=value, ...) instead of ndb.ContextOption(option=value, ...). If you are including your own copy of NDB into your app you are not directly affected, but the above changes will affect you when you upgrade to NDB 0.9.9 (items 1-3) or NDB 1.0.0 (item 4). Note that NDB 1.0.0 is not yet released, but an approximation is available in the sandbox branch of the repo. -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Why can't I filter() based on key?
On Tuesday, March 20, 2012 2:26:20 PM UTC-7, Mike wrote:
>
> I am trying to use the filter() method based on a key argument. For
> example:
>
> user_key="agxkZXZ-emFiZXRhLTJyCgsSBFVzZXIYGAw"
> d=datamodel.Task.all()
> d.filter("user=",user_key)
> results=d.fetch(1024)
> This produces zero results, and its probably because the key isn't in
> the right format. If this where gql would I have to use the Key()
> function like this:
>
> db.gqlQuery("select * from Task where user=Key(:1)",user_key)
> However I don't think I can use the Key() function within a filter,
> but there must be some python functional equivalent... Is there a way
> to properly format a key to be used within the filter() method?
>
> User is defined as:
> user = db.ReferenceProperty(None)
You have to wrap user_key in a call to db.Key(), e.g.
d.filter("user =", db.Key(user_key) )
The user_key value you have is a string but the datastore contains a Key
object, which is a structured thing that will never match any string.
--Guido van Rossum
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/1-foEUHD5q8J.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: equivalent for is_saved() with ndb
On Tuesday, March 27, 2012 6:41:25 AM UTC-7, aschmid wrote: > > is there an equivalent of the db.Model function is_saved() with ndb.Model? No; what are you trying to do? You might be able to check whether the entity has a key and if so, whether that key isn't incomplete: if ent.key and ent.key.id(): # It has a complete key. else: # Hasn't been written, ever. However this can be fooled if you explicitly set the key or the id when you create an entity, e.g. ent = Employee(id='joe') or ent = Employee(key=ndb.Key(Employee, 'joe')) (These two are equivalent.) --Guido van Rossum -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/MeORtJLdaqEJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: equivalent for is_saved() with ndb
You might be able to use get_or_insert() and benefit from the in-memory cache.
On Tue, Mar 27, 2012 at 13:49, Andreas wrote:
> exactly what im doing. i create the keys myself.
>
> rootkey = ndb.Key('root', 'key')
> a = Asset(key=ndb.Key('Asset', 'keyname', parent=rootkey))
>
> a._has_complete_key() # returns true
>
> in my app i work in with entity batches and sometimes it happens that i need
> to create ancestors for that entity which could already exist so till now i
> used is_saved() to filter out the entities that are already saved to the
> datastore to avoid putting them again (or at least trying to put them again)
>
>
> On Mar 27, 2012, at 3:38 PM, Guido van Rossum wrote:
>
> On Tuesday, March 27, 2012 6:41:25 AM UTC-7, aschmid wrote:
>>
>> is there an equivalent of the db.Model function is_saved() with ndb.Model?
>
>
> No; what are you trying to do? You might be able to check whether the entity
> has a key and if so, whether that key isn't incomplete:
>
> if ent.key and ent.key.id():
> # It has a complete key.
> else:
> # Hasn't been written, ever.
>
> However this can be fooled if you explicitly set the key or the id when you
> create an entity, e.g.
>
> ent = Employee(id='joe')
>
> or
>
> ent = Employee(key=ndb.Key(Employee, 'joe'))
>
> (These two are equivalent.)
>
> --Guido van Rossum
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/MeORtJLdaqEJ.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
--
--Guido van Rossum (python.org/~guido)
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: equivalent for is_saved() with ndb
Sorry to hear that. I guess you could manually set a flag on the
parent entity that indicates that you haven't written it yet, and
clear it in a pre_put_hook.
BTW is there any information in the parent entities? Did you know you
can create child entities for which no parent actually exists? All you
need is the parent *key*, it doesn't need to have an entity.
--Guido
On Tue, Mar 27, 2012 at 14:47, Andreas wrote:
> not a good option in my case.
>
> On Mar 27, 2012, at 5:23 PM, Guido van Rossum wrote:
>
>> You might be able to use get_or_insert() and benefit from the in-memory
>> cache.
>>
>> On Tue, Mar 27, 2012 at 13:49, Andreas wrote:
>>> exactly what im doing. i create the keys myself.
>>>
>>> rootkey = ndb.Key('root', 'key')
>>> a = Asset(key=ndb.Key('Asset', 'keyname', parent=rootkey))
>>>
>>> a._has_complete_key() # returns true
>>>
>>> in my app i work in with entity batches and sometimes it happens that i need
>>> to create ancestors for that entity which could already exist so till now i
>>> used is_saved() to filter out the entities that are already saved to the
>>> datastore to avoid putting them again (or at least trying to put them again)
>>>
>>>
>>> On Mar 27, 2012, at 3:38 PM, Guido van Rossum wrote:
>>>
>>> On Tuesday, March 27, 2012 6:41:25 AM UTC-7, aschmid wrote:
>>>>
>>>> is there an equivalent of the db.Model function is_saved() with ndb.Model?
>>>
>>>
>>> No; what are you trying to do? You might be able to check whether the entity
>>> has a key and if so, whether that key isn't incomplete:
>>>
>>> if ent.key and ent.key.id():
>>> # It has a complete key.
>>> else:
>>> # Hasn't been written, ever.
>>>
>>> However this can be fooled if you explicitly set the key or the id when you
>>> create an entity, e.g.
>>>
>>> ent = Employee(id='joe')
>>>
>>> or
>>>
>>> ent = Employee(key=ndb.Key(Employee, 'joe'))
>>>
>>> (These two are equivalent.)
>>>
>>> --Guido van Rossum
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msg/google-appengine/-/MeORtJLdaqEJ.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Google App Engine" group.
>>> To post to this group, send email to google-appengine@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> google-appengine+unsubscr...@googlegroups.com.
>>> For more options, visit this group at
>>> http://groups.google.com/group/google-appengine?hl=en.
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> To unsubscribe from this group, send email to
>> google-appengine+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/google-appengine?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to google-appengine@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
--Guido van Rossum (python.org/~guido)
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: equivalent for is_saved() with ndb
But that's a completely different use case -- and old db's is_saved()
would not cover it. You can probably implement it by overriding
__setattr__ on your model class (if you promise yourself not to modify
repeated properties in-place).
--Guido
On Tue, Mar 27, 2012 at 18:05, Waleed Abdulla wrote:
> I've been looking for a good way to do something similar as well. I load (or
> create) an entity, do some operations on it (that may or may not change the
> entity), and then at the end I need to save it or skip saving it if nothing
> has changed. It would be nice to have something like is_dirty() which is
> True if the entity has never been saved or has changed since it was loaded.
>
>
>
> On Tue, Mar 27, 2012 at 3:19 PM, Guido van Rossum wrote:
>>
>> Sorry to hear that. I guess you could manually set a flag on the
>> parent entity that indicates that you haven't written it yet, and
>> clear it in a pre_put_hook.
>>
>> BTW is there any information in the parent entities? Did you know you
>> can create child entities for which no parent actually exists? All you
>> need is the parent *key*, it doesn't need to have an entity.
>>
>> --Guido
>>
>> On Tue, Mar 27, 2012 at 14:47, Andreas wrote:
>> > not a good option in my case.
>> >
>> > On Mar 27, 2012, at 5:23 PM, Guido van Rossum wrote:
>> >
>> >> You might be able to use get_or_insert() and benefit from the in-memory
>> >> cache.
>> >>
>> >> On Tue, Mar 27, 2012 at 13:49, Andreas wrote:
>> >>> exactly what im doing. i create the keys myself.
>> >>>
>> >>> rootkey = ndb.Key('root', 'key')
>> >>> a = Asset(key=ndb.Key('Asset', 'keyname', parent=rootkey))
>> >>>
>> >>> a._has_complete_key() # returns true
>> >>>
>> >>> in my app i work in with entity batches and sometimes it happens that
>> >>> i need
>> >>> to create ancestors for that entity which could already exist so till
>> >>> now i
>> >>> used is_saved() to filter out the entities that are already saved to
>> >>> the
>> >>> datastore to avoid putting them again (or at least trying to put them
>> >>> again)
>> >>>
>> >>>
>> >>> On Mar 27, 2012, at 3:38 PM, Guido van Rossum wrote:
>> >>>
>> >>> On Tuesday, March 27, 2012 6:41:25 AM UTC-7, aschmid wrote:
>> >>>>
>> >>>> is there an equivalent of the db.Model function is_saved() with
>> >>>> ndb.Model?
>> >>>
>> >>>
>> >>> No; what are you trying to do? You might be able to check whether the
>> >>> entity
>> >>> has a key and if so, whether that key isn't incomplete:
>> >>>
>> >>> if ent.key and ent.key.id():
>> >>> # It has a complete key.
>> >>> else:
>> >>> # Hasn't been written, ever.
>> >>>
>> >>> However this can be fooled if you explicitly set the key or the id
>> >>> when you
>> >>> create an entity, e.g.
>> >>>
>> >>> ent = Employee(id='joe')
>> >>>
>> >>> or
>> >>>
>> >>> ent = Employee(key=ndb.Key(Employee, 'joe'))
>> >>>
>> >>> (These two are equivalent.)
>> >>>
>> >>> --Guido van Rossum
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "Google App Engine" group.
>> >>> To view this discussion on the web visit
>> >>> https://groups.google.com/d/msg/google-appengine/-/MeORtJLdaqEJ.
>> >>> To post to this group, send email to
>> >>> google-appengine@googlegroups.com.
>> >>> To unsubscribe from this group, send email to
>> >>> google-appengine+unsubscr...@googlegroups.com.
>> >>> For more options, visit this group at
>> >>> http://groups.google.com/group/google-appengine?hl=en.
>> >>>
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "Google App Engine" group.
>> >>> To post to this group, send email to
>> >>> google
[google-appengine] Re: NDB Caching Question
On Monday, April 23, 2012 10:21:26 PM UTC-7, Richard Arrano wrote: > > I'm switching from db to ndb and I have a question regarding caching: > > In the old db, I would have a class X that contains a reference to a > class Y. The Y type would be accessed most frequently and rarely > change. So when I would query an X and retrieve the Y type it points > to, I would store X in the memcache with the actual instance Y rather > than the key. If X is invalidated in the memcache, then so is the Y > instance but otherwise I would skip the step of querying Y upon re- > retrieving X from the memcache. Is there any way to do this in ndb? Or > must I re-query each Y type even if it is from memcache or context? > If you leave the caching to NDB, you probably needn't worry about this much. It's going to be an extra API call to retrieve Y (e.g. y = x.yref.get()) but that will generally be a memcache roundtrip. If you are retrieving a lot of Xes in one query, there's a neat NDB idiom to prefetch all the corresponding Ys in one roundtrip: xs = MyModel.query(...).fetch() _ = ndb.get_multi([x.yref for x in xs]) This effectively throws away the ys, but populates them in the context cache. After this, for any x in xs, the call x.yref.get() will use the context cache, which is a Python dict in memory. (Its lifetime is one incoming HTTP request.) You can even postpone waiting for the ys, using an async call: xs = MyModel.query(...).fetch() _ = ndb.get_multi_async([x.yref for x in xs]) Now the first time you reference some x.yref.get() it will block for the get_multi_async() call to complete, and after that all subsequent x.yref.get() calls will be satisfied from memory (no server roundtrip at all). -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/5SCo4vdgdOUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: NDB Caching Question
On Tuesday, April 24, 2012 3:07:35 PM UTC-7, Richard Arrano wrote: > > Thank you for the quick and very informative reply. I wasn't even > aware this was possible with NDB. How would those x.yref.get() calls > show up in AppStats? Or would they at all if it's just pulling it from > memory? > If they pull from memory they don't show up in Appstats at all. Otherwise they'll probably look like a memcache Get possibly followed by a datastore Get. > Thank you Kaan as well, I will actually experiment with the > PickleProperty and see what's faster. I like that solution because the > X kind is not one I expect to be heavily cached so I don't mind > actually caching the pickled instance as I expect them to be evicted > within a relatively short amount of time. > If you're considering storing a pickled entity, you should look into LocalStructuredProperty, which is a little bit more efficient (but doesn't store the key). > I also wanted to ask: I saw someone did a speed test with NDB and I > noticed he was pulling 500 entities of 40K and in the worst-case 0% > cache hit scenario, it took something like 8-10 seconds. I was > actually planning to have a piece of my application regularly query > and cache ~2500 entities(of 2500) and sort on it to avoid a huge > amount of indices(and a NOT IN filter that would really slow things > down). Is this feasible or would you expect his results to scale, i.e. > 500 entities with 0% cache hits * 5 ~= 40-50s in my usage scenario? Or > was there something unique to his situation with his indices and large > amount of data? In mine each entity has about 10 properties with zero > indices. If this is the case I'll probably copy the entities into a > JsonProperty that occasionally gets updated and simply query/cache > that since I don't expect the 2500 entities to change very often. > There are too many unknown variables here. You're best off benchmarking this yourself... -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/sxaizdposEIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: can i select a name for a key when using ndb?
On Saturday, June 16, 2012 9:55:36 AM UTC-7, saintthor wrote: > > for db.expando E: > > E( key_name = 'zzz' ).put() > > will create an entity with a key named 'zzz'. but when using ndb.expando, > it set an attribute named 'key_name ' with the value 'zzz'. > > how to make the ndb work as db? > Use E(id='zzz').put() --Guido van Rossum -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/8alD7GxEKV4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.7.0 Prerelease Now Available
On Tuesday, June 19, 2012 7:22:15 AM UTC-7, Christopher Ramírez wrote:
>
> On Monday, June 18, 2012 6:00:23 PM UTC-6, Marce (Google) wrote:
>>
>> - Projection queries are now supported in NDB.
>>
>
> I have problems with projection queries. If I use them with
> .order('-__key__').fetch(limit=20) records returned seems to be only the
> even or odd ones. I opened a new discussion in this group but it was never
> publicised.
>
To be clear, is this with NDB or with db? (I'm guessing db, since NDB
doesn't support the syntax .order('-__key__') -- you'd have to use
.order(-MyClass._key).
Either way, can you please submit this to stackoverflow, with more details
about what your code looks like and so on, so someone can help you?
--Guido van Rossum
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/Yus00akylQQJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: UnprojectedPropertyError (unseen error until today)
This seems a repeat from a post made yesterday. The solution was a bug in user code that wasn't detected by NDB before; the user write SELECT var instead of SELECT *. On Tuesday, June 19, 2012 2:27:16 PM UTC-7, erineg1 wrote: > > NDB Team-- > > It appears something happened today as I had some code break, in particular > it's this line: > > program_dict["provider_key"] = program.provider.urlsafe() > program.provider is a KeyProperty() > > is yielding: > > UnprojectedPropertyError: Property provider is not in the projection > I'm iterating through the results of a Future: > > nationwide_programs = nationwide_programs_future.get_result() > > I'm completely puzzled. Did something change I should be looking at? > > If so, should I be taking a different approach? > > > Property provider is not in the projection > Traceback (most recent call last): > File > "/base/data/home/apps/s~searchbertha-hrd/81.359741171174247232/models.py", > line 2093, in get_programs > nationwide_programs_dict_list, categories = > summarize_programs(nationwide_programs, categories, sub_categories, > nationwide_searchable_index, indexed) > File > "/base/data/home/apps/s~searchbertha-hrd/81.359741171174247232/models.py", > line 1975, in summarize_programs > program.provider_key = program_dict["provider_key"] = > program.provider.urlsafe() > File > "/base/python_runtime/python_lib/versions/1/google/appengine/ext/ndb/model.py", > line 1238, in __get__ > return self._get_value(entity) > File > "/base/python_runtime/python_lib/versions/1/google/appengine/ext/ndb/model.py", > line 1213, in _get_value > 'Property %s is not in the projection' % (self._name,)) > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/TKKLkifqSNwJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: how to get all entities of one Model for ndb?
On Thursday, June 21, 2012 2:52:01 AM UTC-7, saintthor wrote: > > as db.Model.all() does. > Let me remind people of the "NDB cheat sheet" which answers every conceivable question about how to spell db's patterns in NDB: https://docs.google.com/a/google.com/document/d/1AefylbadN456_Z7BZOpZEXDq8cR8LYu7QgI7bt5V0Iw/edit --Guido van Rossum -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bJ7f0jQC9WAJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: how many db ops does it cost when accessing a ndb entity with a StructuredProperty?
On Sunday, June 24, 2012 5:40:15 AM UTC-7, saintthor wrote: > > are there a more ops for the StructuredProperty? StructuredProperty is essentially just a bunch of properties with funny names. The cost depends on how many indexed properties you have in total in an entity (including in structured properties). So e.g. these two (Flat and Nested) below have the same cost: class Flat(ndb.Model): name = ndb.StringProperty() address_line1 = ndb.StringProperty() address_line2 = ndb.StringProperty() and class Addres(ndb.Model): line1 = ndb.StringProperty() line2 = ndb.StringProperty() class Nested(ndb.Model): name = ndb.StringProperty() address = ndb.StructuredProperty(Address) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/X0VKumjWFqIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: in admin console, how to select a db or ndb entity with gql by id or name?
That has nothing to do with ndb; none of those would work with db either. The correct query has "WHERE __key__ = KEY()". On Sunday, July 1, 2012 5:38:00 PM UTC+2, saintthor wrote: > > ndbModel( id = 'KeyName' ).put() > > SELECT * FROM ndbModel where key_name=KEY( 'dbModel', 'KeyName' ) > SELECT * FROM ndbModel where name=KEY( 'dbModel', 'KeyName' ) > SELECT * FROM ndbModel where key=KEY( 'dbModel', 'KeyName' ) > SELECT * FROM ndbModel where id=KEY( 'dbModel', 'KeyName' ) > > none of the above can get the entity. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/V8ps313xfc4J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: in admin console, how to select a db or ndb entity with gql by id or name?
I'm sorry, I don't understand. What function do you believe is missing in NDB? On Monday, July 2, 2012 11:51:37 AM UTC+2, saintthor wrote: > > thank you. > > and, don't you think ndb need this function too? > > 在 2012年7月2日星期一UTC+8上午3时14分36秒，Guido van Rossum写道： >> >> That has nothing to do with ndb; none of those would work with db either. >> The correct query has "WHERE __key__ = KEY()". >> >> On Sunday, July 1, 2012 5:38:00 PM UTC+2, saintthor wrote: >>> >>> ndbModel( id = 'KeyName' ).put() >>> >>> SELECT * FROM ndbModel where key_name=KEY( 'dbModel', 'KeyName' ) >>> SELECT * FROM ndbModel where name=KEY( 'dbModel', 'KeyName' ) >>> SELECT * FROM ndbModel where key=KEY( 'dbModel', 'KeyName' ) >>> SELECT * FROM ndbModel where id=KEY( 'dbModel', 'KeyName' ) >>> >>> none of the above can get the entity. >>> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/jbx-7LbrVP8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: ndb BadRequestError only on production
On Wednesday, July 4, 2012 1:59:49 AM UTC+2, mma wrote:
>
> Hi there.
>
> I get the following error only on production: BadRequestError: BLOB,
> ENITY_PROTO or TEXT properties must be in a raw_property field
>
> It happens when I put() a instance of the Receipt class (extends ndb.Model)
>
> Below, I attach the model and the handler where the code breaks (only in
> production)
>
>
> class Receipt(RModel):
> ownerId = ndb.IntegerProperty()
> houseId = ndb.IntegerProperty()
> renterId = ndb.IntegerProperty()
> year = ndb.IntegerProperty()
> month_number = ndb.IntegerProperty()
> code = ndb.StringProperty()
> description = ndb.StringProperty()
> value = ndb.StringProperty()
>
> owner = ndb.ComputedProperty(lambda self: Owner.get_by_id(self.ownerId))
> house = ndb.ComputedProperty(lambda self: House.get_by_id(self.houseId))
> renter = ndb.ComputedProperty(lambda self: Renter.get_by_id(self.renterId))
> month = ndb.ComputedProperty(lambda self:
> month_number_to_string(self.month_number))
>
These ComputedProperties look suspicious. The owner, house and renter
lambdas return entities; are you sure you don't mean to return their keys
instead? E.g. Owner.get_by_id(id) loads an Owner entity (which is a
blocking datastore call). If you just want to store the key, you can use
ndb.Key(Owner, id) instead.
If you really want to store these as entities, it's possible that you can
get away by declaring the ComputedProperty as indexed=False. But you're
probably better off declaring them as e.g. StructuredProperty(Owner), and
if you want them filled in automatically, you could do that in a pre-post
hook.
>
> class RModel(ndb.Model):
> created = ndb.DateTimeProperty(auto_now_add = True)
> changed = ndb.DateTimeProperty(auto_now_add = True)
> creatorId = ndb.IntegerProperty()
> changerId = ndb.IntegerProperty()
>
> #def to_dict(self):
> # return ndb.to_dict(self, {'id':self.key().id()})
>
Aside: What are you trying to do here? I think maybe you meant this:
def to_dict(self):
result = super(RModel, self).to_dict()
result['id'] = self.key.id()
return result
???
>
> def set_attributes(self, **attrs):
> props = self.properties()
> for prop in props.values():
> if prop.name in attrs:
> prop.__set__(self, attrs[prop.name])
>
Aside: this looks like code from old db. Its equivalent is
ent.populate(**attrs), except the latter complains if you specify a keyword
that has no corresponding property.
>
>
> class ReceiptNew(BaseHandler):
> def Get(self):
> user_id = self.get_user_id()
> owner = Owner.get_by_id(user_id)
> receipt = Receipt(value="")
> houses = list(House.gql("where ownerId = :1", owner.key.id()))
> renters = list(Renter.gql("where ownerId = :1", owner.key.id()))
> context = {'receipt': receipt, 'houses': houses, 'renters': renters,
> 'new': True}
> self.render_response('receipt-edit.html', **context)
>
> def post(self):
> user_id = self.get_user_id()
> owner = Owner.get_by_id(user_id)
>
> data = {
> 'year': self.request.get('year'),
> 'month': self.request.get('month'),
> 'house': self.request.get('house'),
> 'renter': self.request.get('renter'),
> 'value': self.request.get('value'),
> 'paid': self.request.get('paid')
> }
>
> receipt = Receipt()
> receipt.year = int(data.get('year'))
> receipt.month_number = int(data.get('month'))
> receipt.houseId = int(data.get('house'))
> receipt.renterId = int(data.get('renter'))
> receipt.value = data.get('value')
> receipt.ownerId = owner.key.id()
> receipt.put() # code breaks here, only in production
> self.redirect('/receipts')
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/iGh6NGBKo4wJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Transactions and queries in python app engine
On Monday, July 9, 2012 4:28:30 PM UTC+2, pdknsk wrote: > > I don't know if it's documented, but it was mentioned in the release > notes. > > The Datastore API now includes a NonTransactional decorator to ensure > that a function is run outside of a transaction. Existing transactions > are paused while the function is executing. Also, in NDB you can run multiple transactions as well as non-transactional code "concurrently" using the propagation option to transaction() or @transactional(), or the @non_transactional decorator. (I put "concurrently" in parentheses since this is not a multi-threading API but simply uses NDB's async I/O facilities.) See the docs at https://developers.google.com/appengine/docs/python/ndb/transactions (which sadly misspell this flag as "propogation". I will fix this.) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/9zRp-2R_EaYJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Fwd: Is anyone using Appstats with Firepython support?
There's an obscure feature in Appstats that integrates Firepython support. (See https://developers.google.com/appengine/articles/firepython) It appears Firepython is no longer maintained and I doubt there are many users of the combination. Would anyone be terribly disappointed if we dropped support for Firepython in appstats? -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Creating a new instance of an entity using ndb has previously interacted with entity's data stored
The "default=[]" is causing this. All instances share the same list object
this way. You'll have to find some other way of initializing the value.
Maybe you cna just use a repeated string property?
On Saturday, July 28, 2012 11:56:46 AM UTC-7, Robert Fischer wrote:
>
> Hi,
>
> I'm creating a new instance of a model and it's getting the data from the
> previous model I created.
>
> Any idea what could be happening here?
>
> class Keyword(ndb.Model):
> user_owner = ndb.UserProperty()
> description = ndb.StringProperty(indexed=False)
> time_created = ndb.DateTimeProperty(auto_now_add=True, indexed=False)
> deleted = ndb.BooleanProperty(default=False)
> keyword_list = ndb.StringProperty(repeated=True, indexed=True) #Indexed
> required_keyword_list = ndb.StringProperty(repeated=True,
> indexed=False)
>
> threshold = ndb.IntegerProperty(indexed=False)
>
> last_found_on_id = ndb.StringProperty(indexed=False)
> ids_found_on = ndb.JsonProperty(default=[], indexed=False)
> times_found = ndb.ComputedProperty(lambda self:
> len(self.ids_found_on)) #Indexed
> time_last_found = ndb.DateTimeProperty(indexed=False)
>
> class KeywordHandler(webapp2.RequestHandler)
> def post(self):
> ...
> logging.warning('Creating new keyword object.')
> keyword_to_save = Keyword()
> logging.warning('ids associated with keyword: %s %s %s' %
> (str(keyword_to_save.key), str(keyword_to_save),
> keyword_to_save.ids_found_on))
> ...
>
> The first time I run this it works great:
> WARNING 2012-07-28 18:27:32,403 frontend_keywords.py:176] ids associated
> with keyword: None Keyword() []
>
> When run a second time I get this in my log file -- why is the
> ids_found_on already populated for a new model instance? The ids_found_on
> matches the previously created entity:
>
> WARNING 2012-07-28 18:27:40,459 frontend_keywords.py:176] ids associated
> with keyword: None Keyword()
> ['ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNCAxMzo0NTozOC1zeW1hbnRlYyBuDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNCAxNjozNDo0Ni1kb2xpY2EgcHJvDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNCAxNzowMDowOS10cmVuZG5ldCA4DA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNCAxNzoyNzo1OS00MCBzY2VwdHJlDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNCAxNzozNDoyNi0ycGFjayBjcmFmDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNSAwNzowMDowMy1uZXRnZWFyIHB1DA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNSAwNzoxNjoxMC1waW9uZWVyIHZzDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNSAxNjo0ODo0Mi1jcm9jcyBjb3VwDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNSAxOTowMDo0OS1zYW1zdW5nIDgzDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNSAxOTozMzowOC1tZW5zIHdlYXJoDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNSAxOTo1Mjo1NC10cmFtb250aW5hDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNiAwMTo0MDozOS1jcmFmdHNtYW4gDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNiAxMjoxMjozOC1ib3cgcmFrZSB3DA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNiAxNzo1NDoyOC00MSB2b3JuYWRvDA',
>
> 'ahBkZXZ-ZGVhbHNjb3JjaGVyciwLEghSU1NFbnRyeSIeMjAxMi0wNy0yNiAxODowNDoyOC1sb2dpdGVjaCBoDA']
>
> Thanks,
> Robert
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/F0VkBIsYR2YJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Possible error in the documentation?
I'll just fix this. (If you already filed a bug tell me the URL.)
On Sunday, August 26, 2012 4:30:33 PM UTC-7, Anand Mistry wrote:
>
> Yes. Please file a bug.
>
> On Monday, 27 August 2012 04:05:52 UTC+10, Attila-Mihaly Balazs wrote:
>>
>> I believe that the following example is incorrect in the documentation
>> [1]:
>>
>> app = ndb.toplevel(webapp.WSGIApplication)([('/', MainPage)])
>>
>> It sees to me that the correct way to write this would be:
>>
>> app = ndb.toplevel(webapp.WSGIApplication([('/', MainPage)]))
>>
>> Otherwise it just wraps the class, not the actual instance of it and
>> while it seems to work, it can lead to silent corruptions (dropping of
>> async tasks).
>>
>> Am I correct in this assumption? Should I file a bug for this?
>>
>> Thanks,
>> Attila
>>
>> [1] https://developers.google.com/appengine/docs/python/ndb/async
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/google-appengine/-/Z96C3-j9sRwJ.
To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Unit testing with webtest problem
You'll probably get more help from StackOverflow.com. You'll need to provide more info, nobody can help you debug this with just that traceback information unless they're psychic. On Sunday, September 9, 2012 10:06:33 AM UTC-7, Richard Arrano wrote: > > Hello, > I've been using webtest to unit test my application and I've encountered a > strange issue. I wrap many of my get/post handlers in decorators and in > some of those decorators, I put ndb.Model instances in os.environ for later > use in the subsequent handler. This works on my local dev server and in > production. However, when I run nosetests it always gives me an error: > > os.environ["user"] = user > File "C:\Python27\lib\os.py", line 420, in __setitem__ > putenv(key, item) > TypeError: must be string, not User > > Any ideas on how to mitigate this so my tests won't error out at this > point? > > Thanks, > Richard > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/ML4cFtd3TDUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: NDB Query offset - limit
On Wednesday, September 19, 2012 8:51:48 AM UTC-7, Moises Belchin wrote: > Hi guys, > > I have this query: > > q = MyKind.query() > regs = q.fetch(offset = 990, limit = 10) // *Returns 10 entities.* > > If I press next button on my UI: > > q = MyKind.query() > regs = q.fetch(offset = 1000, limit = 10) // *Returns 0 * *entities* *.* > > MyKind has 1300 entities. > > Any thoughts? > > Thanks and regards. > Moisés Belchín. > I can confirm this. I believe it is intentional to encourage you to use cursors instead of offsets for such queries; internally, you are paying for reading all the entities that you are skipping using the offset. --Guido -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/uNrSdb1M29wJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: NDB Query offset - limit
On Thu, Sep 20, 2012 at 1:59 AM, Moises Belchin wrote: > Hi Guido, > > Thanks for your answer. > > We test to use query cursors and we decide don't use them because query > cursors don't work in reverse order if you don't do query orders. If you have inequalities the order you give must start with the property involved in the inequalities (<, <=, >, >=, !=). > We have queries with many filters and we can't do query orders for this > queries. In this case if we use cursors, the reverse function doesn't work > properly. Did you follow the recipe for paging backwards in the docs? https://developers.google.com/appengine/docs/python/ndb/queries#cursors > We want to know how we must work in this cases without create indexes for > this kind of queries. Can you post your model and query here? (Or mail them privately if you think they're too sensitive.) > Thanks and regards. > Moisés Belchín. > > > > 2012/9/20 Guido van Rossum >> >> On Wednesday, September 19, 2012 8:51:48 AM UTC-7, Moises Belchin wrote: >>> >>> Hi guys, >>> >>> I have this query: >>> >>> q = MyKind.query() >>> regs = q.fetch(offset = 990, limit = 10) // Returns 10 entities. >>> >>> If I press next button on my UI: >>> >>> q = MyKind.query() >>> regs = q.fetch(offset = 1000, limit = 10) // Returns 0 entities . >>> >>> MyKind has 1300 entities. >>> >>> Any thoughts? >>> >>> Thanks and regards. >>> Moisés Belchín. >> >> >> I can confirm this. I believe it is intentional to encourage you to use >> cursors instead of offsets for such queries; internally, you are paying for >> reading all the entities that you are skipping using the offset. >> >> --Guido >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/uNrSdb1M29wJ. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: NDB Transactions
Sorry, the underlying Datastore does not (yet) support nested transactions, so you cannot nest them at all. On Monday, September 24, 2012 1:53:43 AM UTC-7, Moises Belchin wrote: > > Hi, > > In NDB we can use @ndb.transactional(xg=True). > > @ndb.transactional(xg=True) can be nested? If yes, how many > @ndb.transactional(xg=True) > can you nested? > > Thanks and regards. > Moisés Belchín. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/WD_BxSS2Bo8J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: NDB Query offset - limit
Hi Moises, I think you may be able to solve this by adding a sort order by key, as follows: q_next = q.order(MaqHistMov.idsm, MaqHistMov.key) q_previous = q.order(-MaqHistMov.idsm, -MaqHistMov.key) FWIW, here is an example that I think represents what you are doing: class M(ndb.Model): a = ndb.IntegerProperty() b = ndb.IntegerProperty() M(a=1, b=1).put() M(a=2, b=1).put() M(a=3, b=1).put() M(a=4, b=2).put() M(a=5, b=3).put() M(a=6, b=4).put() q = M.query() q_asc = q.order(M.b, M.key) q_desc = q.order(-M.b, -M.key) r1, c1, m1 = q_asc.fetch_page(3) print [r.a for r in r1] r2, c2, m2 = q_asc.fetch_page(3, start_cursor=c1) print [r.a for r in r2] r3, c3, m3 = q_desc.fetch_page(3, start_cursor=c2.reversed()) print [r.a for r in r3] r4, c4, m4 = q_desc.fetch_page(3, start_cursor=c3) print [r.a for r in r4] This outputs the following for me, which I think is correct: [1, 2, 3] [4, 5, 6] [6, 5, 4] [3, 2, 1] Note that the final query does not reverse the cursor, since its start cursor is already reversed. So you only reverse the cursor upon reversing query directions. --Guido On Mon, Sep 24, 2012 at 4:12 AM, Moises Belchin wrote: > Hi Guido, > > Thanks in advance to take your time and figure out our problem. > > I send you the code we're using and an image to show you our model and some > entities to review our query results. > > The first problem we encountered is that idsm property must be indexed if > you want to order by MaqHistMov.idsm desc. If you use limit, offset query > fetch this requirement doesn't exist. -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: NDB Query offset - limit
FWIW, this behavior appears to be a bug in NDB. I've filed http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=210 I'll investigate. On Wed, Sep 19, 2012 at 4:22 PM, Guido van Rossum wrote: > On Wednesday, September 19, 2012 8:51:48 AM UTC-7, Moises Belchin wrote: >> >> Hi guys, >> >> I have this query: >> >> q = MyKind.query() >> regs = q.fetch(offset = 990, limit = 10) // Returns 10 entities. >> >> If I press next button on my UI: >> >> q = MyKind.query() >> regs = q.fetch(offset = 1000, limit = 10) // Returns 0 entities . >> >> MyKind has 1300 entities. >> >> Any thoughts? >> >> Thanks and regards. >> Moisés Belchín. > > > I can confirm this. I believe it is intentional to encourage you to use > cursors instead of offsets for such queries; internally, you are paying for > reading all the entities that you are skipping using the offset. > > --Guido -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: NDB Query offset - limit
On Tue, Sep 25, 2012 at 1:12 AM, Moises Belchin wrote: > Hi Guido, > > Thanks for your comments and help. > > I think adding sort order by key is a problem. For example: > > If you have a kind on which you have equals filters using fetch with limit, > offset you don't need to use indexes. However if you use fetch_page, you > need to create indexes for q_desc. If you have a big app and use indexes for > any kind you'll quickly consume your index quotas. OTOH the cost of using offset is not zero either -- you pay as much for the skipped entities as for the results. > In addition, this result, > > [1, 2, 3] > [4, 5, 6] > [6, 5, 4] > [3, 2, 1] > > for me is not correct. > > I think if you show a grid to an user (1,2,3), the user clicks on next > button and he views (4,5,6). Then If the user now clicks on previous button > the user must view (1,2,3) instead of (3,2,1). > > Now I you use fetch with offset and limit you get this behaviour(1,2,3)-> > (4,5,6) <- (1,2,3), however with fetch_page and reversed cursor this not. If you want to display the results as [3, 2, 1] just reverse the results list in memory. [...].reverse() does the job in-line in O(N) time with o extra space needed. > I think, one option may be use fetch_page with positive or negative limit to > simulate this behaviour. I don't know If this can be correct. If you're proposing a feature change, underneath the implementation would have to do all the same work. Consider fetch_page() a building block that makes it possible to do the right thing, not necessarily the most convenient thing (which would depend on the needs of a particular app). > Thanks for your time and for heard me. > Regards. > Moisés Belchín. > > > > 2012/9/24 Guido van Rossum >> >> Hi Moises, >> >> I think you may be able to solve this by adding a sort order by key, as >> follows: >> >> q_next = q.order(MaqHistMov.idsm, MaqHistMov.key) >> q_previous = q.order(-MaqHistMov.idsm, -MaqHistMov.key) >> >> FWIW, here is an example that I think represents what you are doing: >> >> class M(ndb.Model): >> a = ndb.IntegerProperty() >> b = ndb.IntegerProperty() >> M(a=1, b=1).put() >> M(a=2, b=1).put() >> M(a=3, b=1).put() >> M(a=4, b=2).put() >> M(a=5, b=3).put() >> M(a=6, b=4).put() >> q = M.query() >> q_asc = q.order(M.b, M.key) >> q_desc = q.order(-M.b, -M.key) >> r1, c1, m1 = q_asc.fetch_page(3) >> print [r.a for r in r1] >> r2, c2, m2 = q_asc.fetch_page(3, start_cursor=c1) >> print [r.a for r in r2] >> r3, c3, m3 = q_desc.fetch_page(3, start_cursor=c2.reversed()) >> print [r.a for r in r3] >> r4, c4, m4 = q_desc.fetch_page(3, start_cursor=c3) >> print [r.a for r in r4] >> >> This outputs the following for me, which I think is correct: >> >> [1, 2, 3] >> [4, 5, 6] >> [6, 5, 4] >> [3, 2, 1] >> >> Note that the final query does not reverse the cursor, since its start >> cursor is already reversed. So you only reverse the cursor upon >> reversing query directions. >> >> --Guido >> >> On Mon, Sep 24, 2012 at 4:12 AM, Moises Belchin >> wrote: >> > Hi Guido, >> > >> > Thanks in advance to take your time and figure out our problem. >> > >> > I send you the code we're using and an image to show you our model and >> > some >> > entities to review our query results. >> > >> > The first problem we encountered is that idsm property must be indexed >> > if >> > you want to order by MaqHistMov.idsm desc. If you use limit, offset >> > query >> > fetch this requirement doesn't exist. >> >> -- >> --Guido van Rossum (python.org/~guido) > > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: ndb BadRequestError only on production
I don't think this has anything to do with the previous thread that had the
same subject.
The problem must be that your ComputedProperty is trying returning a Model
instance. That's unfortunately not supported; ComputedProperty as it is
currently implemented can only support the "basic" data types like int, str
and the like. I'm not sure why this doesn't trigger an error in the dev
appserver for you -- when I try something similar it does trigger the error
in a test.(*) I'm also not sure what's the best solution; you may just have
to break up the computed structured property up into two separate computed
property.
(*) My session log:
>>> class A(Model):
... x = IntegerProperty()
...
>>> class M(Model):
... a = ComputedProperty(lambda self: A(x=1))
...
>>> m = M()
>>> m.a
A(x=1)
>>> m.put()
WARNING:root:suspended generator _put_tasklet(context.py:274) raised
BadRequestError(BLOB, ENITY_PROTO or TEXT property a must be in a
raw_property field)
WARNING:root:suspended generator put(context.py:703) raised
BadRequestError(BLOB, ENITY_PROTO or TEXT property a must be in a
raw_property field)
Traceback (most recent call last):
File "", line 1, in
File "/Users/guido/appengine-ndb-experiment/ndb/model.py", line 3151, in
_put
return self._put_async(**ctx_options).get_result()
File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 325,
in get_result
self.check_success()
File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 368,
in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/Users/guido/appengine-ndb-experiment/ndb/context.py", line 703, in
put
key = yield self._put_batcher.add(entity, options)
File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 368,
in _help_tasklet_along
value = gen.throw(exc.__class__, exc, tb)
File "/Users/guido/appengine-ndb-experiment/ndb/context.py", line 274, in
_put_tasklet
keys = yield self._conn.async_put(options, datastore_entities)
File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 454,
in _on_rpc_completion
result = rpc.get_result()
File
"/usr/local/google_appengine/google/appengine/api/apiproxy_stub_map.py",
line 604, in get_result
return self.__get_result_hook(self)
File
"/usr/local/google_appengine/google/appengine/datastore/datastore_rpc.py",
line 1569, in __put_hook
self.check_rpc_success(rpc)
File
"/usr/local/google_appengine/google/appengine/datastore/datastore_rpc.py",
line 1224, in check_rpc_success
raise _ToDatastoreError(err)
google.appengine.api.datastore_errors.BadRequestError: BLOB, ENITY_PROTO or
TEXT property a must be in a raw_property field
>>>
--Guido
On Monday, October 8, 2012 9:40:49 AM UTC-7, Primijos wrote:
>
> Hi,
>
> I'm getting also this error trying to put() [to be exact, trying to do a
> get_or_insert] an entity with a computed property. My Model looks more or
> less like this (removed non-relevant code):
>
> class DecimalProperty(ndb.StringProperty):
>def _validate(self,value):
> if not isinstance(value,(int,long,basestring,decimal.Decimal)):
> raise ndb.BadValueError("Property %s must be a decimal, string,
> float or int." % self.name)
> return decimal.Decimal(value).quantize(TWOPLACES)
>
>def _to_base_type(self,value):
> if value>=0:
> return "P%010d" % (value*100)
> else:
> new_value = MAX_DECIMAL + (value*100)
> return "N%010d" % new_value
>
>
>def _from_base_type(self,value):
> if value[0] == "P":
> return (decimal.Decimal(value[1:])/100).quantize(TWOPLACES)
> else:
> tmp = decimal.Decimal(value[1:])
> tmp = - (MAX_DECIMAL - tmp)
> return (decimal.Decimal(tmp)/100).quantize(TWOPLACES)
>
>
> class Accounting(ndb.Model):
>expense = DecimalProperty(required=False,indexed=False,
> default=DEC_ZERO)
>investment= DecimalProperty(required=False,indexed=False,
> default=DEC_ZERO)
>
>
> def accountable_basic_compute_balance(accountable):
>expense = accountable.commited.expense - accountable.invoiced.expense
>investment = accountable.invoiced.investment - accountable.invoiced.
> investment
>return Accounting(expense = expense, investment = investment)
>
>
> class AccountableBasic(ndb.Model):
>def __init__(self, *args, **kwargs):
> # initialize structure
> super(AccountableBasic, self).__init__(*args, **kwargs)
> self.commited = Accounting()
> self.invoiced = Accounting()
>
>commited = ndb.StructuredProperty(Accounting, repeated = False, indexed
> =True)
>invoiced = ndb.StructuredProperty(Accounting, repeated = False, indexed
> =True)
>balance = ndb.ComputedProperty(accountable_basic_compute_balance,repeated
> = False, indexed=True)
>
>
> class YearAccounting(AccountableBasic):
># id (key_name) = year
>pass
>
> Everything works fine in the local dev server, when I try to perform a
[google-appengine] Re: ndb BadRequestError only on production
On Monday, October 8, 2012 1:28:57 PM UTC-7, Guido van Rossum wrote:
> I don't think this has anything to do with the previous thread that had
> the same subject.
Apologies, I think you had the same issue, you just got there a different
way.
> The problem must be that your ComputedProperty is trying returning a Model
> instance. That's unfortunately not supported; ComputedProperty as it is
> currently implemented can only support the "basic" data types like int, str
> and the like. I'm not sure why this doesn't trigger an error in the dev
> appserver for you -- when I try something similar it does trigger the error
> in a test.(*) I'm also not sure what's the best solution; you may just have
> to break up the computed structured property up into two separate computed
> property.
>
> (*) My session log:
>
> >>> class A(Model):
> ... x = IntegerProperty()
> ...
> >>> class M(Model):
> ... a = ComputedProperty(lambda self: A(x=1))
> ...
> >>> m = M()
> >>> m.a
> A(x=1)
> >>> m.put()
> WARNING:root:suspended generator _put_tasklet(context.py:274) raised
> BadRequestError(BLOB, ENITY_PROTO or TEXT property a must be in a
> raw_property field)
> WARNING:root:suspended generator put(context.py:703) raised
> BadRequestError(BLOB, ENITY_PROTO or TEXT property a must be in a
> raw_property field)
> Traceback (most recent call last):
> File "", line 1, in
> File "/Users/guido/appengine-ndb-experiment/ndb/model.py", line 3151, in
> _put
> return self._put_async(**ctx_options).get_result()
> File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 325,
> in get_result
> self.check_success()
> File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 368,
> in _help_tasklet_along
> value = gen.throw(exc.__class__, exc, tb)
> File "/Users/guido/appengine-ndb-experiment/ndb/context.py", line 703,
> in put
> key = yield self._put_batcher.add(entity, options)
> File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 368,
> in _help_tasklet_along
> value = gen.throw(exc.__class__, exc, tb)
> File "/Users/guido/appengine-ndb-experiment/ndb/context.py", line 274,
> in _put_tasklet
> keys = yield self._conn.async_put(options, datastore_entities)
> File "/Users/guido/appengine-ndb-experiment/ndb/tasklets.py", line 454,
> in _on_rpc_completion
> result = rpc.get_result()
> File
> "/usr/local/google_appengine/google/appengine/api/apiproxy_stub_map.py",
> line 604, in get_result
> return self.__get_result_hook(self)
> File
> "/usr/local/google_appengine/google/appengine/datastore/datastore_rpc.py",
> line 1569, in __put_hook
> self.check_rpc_success(rpc)
> File
> "/usr/local/google_appengine/google/appengine/datastore/datastore_rpc.py",
> line 1224, in check_rpc_success
> raise _ToDatastoreError(err)
> google.appengine.api.datastore_errors.BadRequestError: BLOB, ENITY_PROTO
> or TEXT property a must be in a raw_property field
> >>>
>
Further experimentation suggests that making the ComputedEntity
indexed=False will make this (seem to) work. However it will behave more
like a LocalStructuredProperty than a StructuredProperty -- the Account
entity is serialized as a single bytestring, not stored as two separate
properties with a dot in their names like the StructuredProperty(Account)
properties are. I'm not sure if you're happy with that or not.
>
> --Guido
>
> On Monday, October 8, 2012 9:40:49 AM UTC-7, Primijos wrote:
>>
>> Hi,
>>
>> I'm getting also this error trying to put() [to be exact, trying to do a
>> get_or_insert] an entity with a computed property. My Model looks more or
>> less like this (removed non-relevant code):
>>
>> class DecimalProperty(ndb.StringProperty):
>>def _validate(self,value):
>> if not isinstance(value,(int,long,basestring,decimal.Decimal)):
>> raise ndb.BadValueError("Property %s must be a decimal, string,
>> float or int." % self.name)
>> return decimal.Decimal(value).quantize(TWOPLACES)
>>
>>def _to_base_type(self,value):
>> if value>=0:
>> return "P%010d" % (value*100)
>> else:
>> new_value = MAX_DECIMAL + (value*100)
>> return "N%010d" % new_value
>>
>>
>>def _from_base_type(self,value):
>> if value[0] == "P":
>> return (decimal.Decimal(value[1:
[google-appengine] Using ThreadManager
No need for threads. Use the async urlfetch API instead. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Z9Fl1QmJs0YJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: ndb, deadlock waiting for flush, and maximum recursion in app_logging
Hi ckhan, I have debugged a similar problem for an internal app once. The root cause was that they were mixing synchronous and async API calls. In particular, inside a tasklet they were making a synchronous call. The synchronous call is run by invoking the event loop recursively; what may happen in this case is that another tasklet is spawned that does the same thing, and so on, until you have many recursive event loop invocations stacked on top of each other. The solution is to find the synchronous call(s) in your tasklet(s) and replace them with "yield ". If this is indirect, e.g. your tasklet calls one of your own non-tasklet helper functions which makes the synchronous call, you will have to convert the helper into a tasklet too. Good luck! It shouldn't be too hard to track this down. --Guido On Monday, October 29, 2012 12:31:55 PM UTC-7, ckhan wrote: > > Hello. I'm having trouble reading an exception log on my app. > > Quick background: > - request was made by taskqueue > - request creates several dozen tasklets that have this code path: > (module/function names simplified): > > mycode3.py:234 myFunc3() > --> mycode2.py:64 myFunc2() > --> mycode1.py:37 myFunc1 > > myFunc1 is yielding on async urlfetch; line 37 is the > @ndb.tasklet decorator > > - taskqueue was configured to re-try 2 more times, which it > did, each failing in exactly the same way > > Actual log: > > 2012-10-29 08:36:59.649 initial generator myFunc1(mycode1.py:37) raised > RuntimeError(maximum recursion depth exceeded) > W 2012-10-29 08:36:59.654 initial generator myFunc1(mycode1.py:37) raised > RuntimeError(maximum recursion depth exceeded) > W 2012-10-29 08:36:59.655 suspended generator myFunc2(mycode2.py:64) > raised RuntimeError(maximum recursion depth exceeded) > W 2012-10-29 08:36:59.665 initial generator myFunc1(mycode1.py:37) raised > RuntimeError(maximum recursion depth exceeded) > W 2012-10-29 08:36:59.669 initial generator myFunc1(mycode1.py:37) raised > RuntimeError(maximum recursion depth exceeded) > W 2012-10-29 08:36:59.670 suspended generator myFunc2(mycode2.py:64) > raised RuntimeError(maximum recursion depth exceeded) > W 2012-10-29 08:36:59.674 initial generator myFunc1(mycode1.py:37) raised > RuntimeError(maximum recursion depth exceeded) > W 2012-10-29 08:36:59.674 suspended generator myFunc2(mycode2.py:64) > raised RuntimeError(maximum recursion depth exceeded) > W 2012-10-29 08:37:00.658 suspended generator myFunc3(mycode3:234) raised > RuntimeError(maximum recursion depth exceeded) > E 2012-10-29 08:37:04.055 > Exception: Deadlock waiting for __call__(_webapp25.py:712) for tasklet flush(context.py:239); pending> > Traceback (most recent call last): > File > "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", > > line 712, in __call__ > handler.post(*groups) > File > "/base/python_runtime/python_lib/versions/1/google/appengine/ext/ndb/tasklets.py", > > line 1050, in add_context_wrapper > ctx.flush().check_success() > File > "/base/python_runtime/python_lib/versions/1/google/appengine/ext/ndb/tasklets.py", > > line 322, in check_success > raise self._exception.__class__, self._exception, self._traceback > RuntimeError: Deadlock waiting for __call__(_webapp25.py:712) for tasklet flush(context.py:239); pending> > E 2012-10-29 08:37:04.086 Traceback (most recent call last): > E 2012-10-29 08:37:04.086 File > "/base/python_runtime/python_lib/versions/1/google/appengine/api/app_logging.py", > > line 98, in emit > E 2012-10-29 08:37:04.086 Traceback (most recent call last): > E 2012-10-29 08:37:04.086 File > "/base/python_runtime/python_lib/versions/1/google/appengine/api/app_logging.py", > > line 91, in emit > E 2012-10-29 08:37:04.086 Traceback (most recent call last): > E 2012-10-29 08:37:04.086 File > "/base/python_runtime/python_lib/versions/1/google/appengine/api/app_logging.py", > > line 98, in emit > E 2012-10-29 08:37:04.086 Traceback (most recent call last): > E 2012-10-29 08:37:04.086 File > "/base/python_runtime/python_lib/versions/1/google/appengine/api/app_logging.py", > > line 91, in emit > E 2012-10-29 08:37:04.086 Traceback (most recent call last): > E 2012-10-29 08:37:04.086 File > "/base/python_runtime/python_lib/versions/1/google/appengine/api/app_logging.py", > > line 98, in emit > > The warnings for the generators would appear to be from 9 separate, > otherwise unrelated tasklets that are all executing concurrently and > all happen to be at different points in the code path. Is that > correct? > > The recursion depth error seems to come from the app_logging. But is > that the underlying cause, or the symptom of the failure? If the > latter, how do I narrow what got me into this situation? > > Any help much appreciated. > -ckhan > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view
Re: [google-appengine] Re: ndb, deadlock waiting for flush, and maximum recursion in app_logging
On Tue, Oct 30, 2012 at 2:54 PM, ckhan wrote: > > Hi Guido - > > Thanks, this was exactly the case! > I was indeed mixing sync/async calls in a tasklet. Glad it's been diagnosed. > A few follow-up questions: > > 1. Is there anything I could have/should have done to make it easier for >me realize this was the problem? Even just showing more frames from >the recursive stack might have clued me. Hardly. This is an extremely esoteric problem -- I have only ever seen it reported once before, but a super high performance internal app. I am still struggling with what would be the best way to diagnose this automatically. We have http://code.google.com/p/appengine-ndb-experiment/issues/detail?id=167 to track this -- maybe you could star it. > 2. Just to clarify my understanding: is the issue specifically making >synchronous calls in a tasklet; Yes. > or is any mixing sync/async in the >same function dangerous? No. >More concretely, is this pattern safe: > > def munch_stuff(): > # synchronously fetch some entities > stuff = ndb.get_multi(...) > > # process each concurrently with munch tasklet > result = yield [munch(c) for c in candidates] That's not safe, assuming you forgot to show the @ndb.tasklet decorator (otherwise, how could the yield have worked). > 2b. If the above is *not* safe, and I need to change the first line to: > >stuff = yield ndb.get_multi_async(...) > >then how do I use munch_stuff in conjunction with 'deferred'? > >It would seem that marking it with a toplevel/tasklet decorator isn't >enough (deferred complains about failing to pickle the function). > >Or, would this configuration be unsupported by deferred? Oh, deferred. It has too many arbitrary limitations. The best thing to do is to only ever use it with simple global functions that only takes simple types (e.g. ints and strings, maybe lists of those) as arguments. > Thanks again! > -ckhan > > > On Tuesday, October 30, 2012 12:07:42 PM UTC-7, Guido van Rossum wrote: >> >> Hi ckhan, >> >> I have debugged a similar problem for an internal app once. The root cause >> was that they were mixing synchronous and async API calls. In particular, >> inside a tasklet they were making a synchronous call. The synchronous call >> is run by invoking the event loop recursively; what may happen in this case >> is that another tasklet is spawned that does the same thing, and so on, >> until you have many recursive event loop invocations stacked on top of each >> other. >> >> The solution is to find the synchronous call(s) in your tasklet(s) and >> replace them with "yield ". If this is indirect, e.g. your >> tasklet calls one of your own non-tasklet helper functions which makes the >> synchronous call, you will have to convert the helper into a tasklet too. >> >> Good luck! It shouldn't be too hard to track this down. >> >> --Guido >> >> On Monday, October 29, 2012 12:31:55 PM UTC-7, ckhan wrote: >>> >>> Hello. I'm having trouble reading an exception log on my app. >>> >>> Quick background: >>> - request was made by taskqueue >>> - request creates several dozen tasklets that have this code path: >>> (module/function names simplified): >>> >>> mycode3.py:234 myFunc3() >>> --> mycode2.py:64 myFunc2() >>> --> mycode1.py:37 myFunc1 >>> >>> myFunc1 is yielding on async urlfetch; line 37 is the >>> @ndb.tasklet decorator >>> >>> - taskqueue was configured to re-try 2 more times, which it >>> did, each failing in exactly the same way >>> >>> Actual log: >>> >>> 2012-10-29 08:36:59.649 initial generator myFunc1(mycode1.py:37) raised >>> RuntimeError(maximum recursion depth exceeded) >>> W 2012-10-29 08:36:59.654 initial generator myFunc1(mycode1.py:37) raised >>> RuntimeError(maximum recursion depth exceeded) >>> W 2012-10-29 08:36:59.655 suspended generator myFunc2(mycode2.py:64) >>> raised RuntimeError(maximum recursion depth exceeded) >>> W 2012-10-29 08:36:59.665 initial generator myFunc1(mycode1.py:37) raised >>> RuntimeError(maximum recursion depth exceeded) >>> W 2012-10-29 08:36:59.669 initial generator myFunc1(mycode1.py:37) raised >>> RuntimeError(maximum recursion depth exceeded) >>> W 2012-10-29 08:36:59.670 suspended generator myFunc2(mycode2.py:64) >>> raised RuntimeError(maximum recursion depth exceeded) >>> W 2012-10-29 08:36:59.674
Re: [google-appengine] Re: ndb, deadlock waiting for flush, and maximum recursion in app_logging
On Thu, Nov 1, 2012 at 9:38 AM, ckhan wrote: > So sorry, I botched my example. Let me try again: munch_stuff is > explicitly *not* a tasklet (no decorator), but 'munch' is, > and I call it with Future's wait_all function: > > def munch_stuff(): > # synchronously fetch some entities > stuff = ndb.get_multi(...) > > # process each concurrently with munch tasklet > ndb.Future.wait_all([munch(s) for s in stuff]) > > Would this construct still be safe re the rule against mixing > sync/async, but let me "munch" concurrently? This example looks fine -- however if you were to call much_stuff() from a tasklet, you'd be in trouble again. > Thanks so much taking the time to answer these questions; > extremely helpful and greatly appreciated. You're welcome. -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Google app engine and lock-in
+1 from me too! On Thu, Dec 6, 2012 at 10:30 AM, Randy Shoup wrote: > From the App Engine side, we're excited to see these several open source > implementations of the App Engine APIs. Keep up the great work. Can't > wait to see what's next. > > Take care, > -- Randy > > Randy Shoup > Director, Google App Engine > > On Wednesday, December 5, 2012 2:41:49 AM UTC-8, Ales Justin wrote: >> >> As promised, and a bit of pr ... >> * >> http://in.relation.to/**Bloggers/FirstCapeDwarfRelease<http://in.relation.to/Bloggers/FirstCapeDwarfRelease> >> >> -Ales >> >> On Nov 23, 2012, at 11:54 PM, Ales Justin wrote: >> >> Sounds interesting... Any idea about the costs ? >> >> >> Deployment and setup wise there are no costs, as it's all open source, >> hence you can build and maintain it yourself. Not that I suggest that -- >> we make our $ out of support. :-) >> >> But obviously if you need any serious environment, you'll need some >> hardware. >> >> But a note of warning, this is a new project, hence not yet around at all >> - just about to do initial release, >> and we definitely welcome community participation. >> >> Otoh, we take big pride in testing things, and we do try to think >> scalable all the time. >> Things are implemented to run in a cluster / cloud, while we admit there >> are some design issues we need to re-evaluate. >> >> I'll push some announcement blog on Monday, >> with more detailed instructions and binaries on how to use things. >> >> I suggest you give it a spin and let us know. >> >> -Ales >> >> On Thursday, November 22, 2012 9:12:44 PM UTC+1, Ales Justin wrote: >> >>> Wrt lock-in, at JBoss/RedHat we're working on this new project: >>> * http://www.jboss.org/**capedwarf <http://www.jboss.org/capedwarf> >>> >>> I'm just about to do initial Beta1 release in the next few days. >>> Perhaps give it a spin and let us know any feedback. >>> >>> -Ales >>> >>> On Wednesday, November 21, 2012 5:26:11 AM UTC+1, Emmanuel Mayssat wrote: >>>> >>>> I am aware of AppScale >>>> http://appscale.cs.ucsb.edu/ >>>> >>>> But is the backend (database) of google app engine also supported? >>>> If so which one is it? >>>> (AppScale support memcachedb, cassandra, mysql, mongodb, and a few >>>> others) >>>> http://www.youtube.com/watch?**v=ebnh3D79xUs<http://www.youtube.com/watch?v=ebnh3D79xUs> >>>> >>>> Regards, >>>> E >>>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit https://groups.google.com/d/** >> msg/google-appengine/-/**vA0Fwjx-qN8J<https://groups.google.com/d/msg/google-appengine/-/vA0Fwjx-qN8J> >> . >> To post to this group, send email to google-a...@googlegroups.**com. >> To unsubscribe from this group, send email to google-appengi...@** >> googlegroups.com. >> For more options, visit this group at http://groups.google.com/** >> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en> >> . >> >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/DOZ7H7Q8WYgJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Guido leaving App Engine team (and Google)
Hey, I'm still here! :-) App Engine is a great product and we are parting as the best of friends. The APp Engine team has a ton of Python talent on board and I am sure they will do a great job carrying the torch. (And occasionally they will be able to do things that I would not have approved of, cranky old man that I am. :-) --Guido On Fri, Dec 7, 2012 at 11:12 AM, pdknsk wrote: > Hopefully this isn't a bad sign for App Engine. > > https://tech.dropbox.com/2012/12/welcome-guido/ > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Sudden increase in Datastore READ operations
There's an experimental feature Appstats Analytics that can help with this. I think the co is all in the SDK but you have to ask a devrel or PM for the docs. Maybe Takashi can help you with this. On Thursday, December 27, 2012, Aswath Satrasala wrote: > Hi, > Suddenly, I am seeing a lot of Datastore read operations. > > I would like to see the details of reads and writes by KINDS during a > certain period. Is there a way to do this? > > Regards > -Aswath > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to > google-appengine@googlegroups.com 'google-appengine@googlegroups.com');> > . > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com 'google-appengine%2bunsubscr...@googlegroups.com');>. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Understanding keys-only query costs
On Fri, Jan 4, 2013 at 1:59 PM, Ryan Chazen wrote: > That link - /_ah/stats/shell - does not seem to be working for me on either > the dev server or the live server. I get a 404 error. Does /_ah/stats work for you? If it doesn't work, you haven't enabled Appstats in your app.yaml (it's in the builtins section). If that does work, you must configure the appstats shell in your appengine_config.py for it to work in production, but it should be on by default in the dev server. (Python only.) -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] AttributeError: 'Entity' object has no attribute '_to_pb' when trying to finalize a blobstore file.
are/google_appengine/google/appengine/api/apiproxy_rpc.py", > line 156, in _WaitImpl > self.request, self.response) > File > "/home/rye/local/share/google_appengine/google/appengine/api/apiproxy_stub.py", > line 125, in MakeSyncCall > method(request, response) > File > "/home/rye/local/share/google_appengine/google/appengine/api/files/file_service_stub.py", > line 652, in _Dynamic_Close > self.open_files[filename].finalize() > File > "/home/rye/local/share/google_appengine/google/appengine/api/files/file_service_stub.py", > line 596, in finalize > datastore.Put(blob_info) > File > "/home/rye/local/share/google_appengine/google/appengine/api/datastore.py", > line 579, in Put > return PutAsync(entities, **kwargs).get_result() > File > "/home/rye/local/share/google_appengine/google/appengine/api/datastore.py", > line 556, in PutAsync > return _GetConnection().async_put(config, entities, local_extra_hook) > File > "/home/rye/local/share/google_appengine/google/appengine/datastore/datastore_rpc.py", > line 1542, in async_put > pbs = [self.__adapter.entity_to_pb(entity) for entity in entities] > File > "/home/rye/local/share/google_appengine/google/appengine/ext/ndb/model.py", > line 562, in entity_to_pb > pb = ent._to_pb() > AttributeError: 'Entity' object has no attribute '_to_pb' > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/EIhPvR7Q2IUJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Problems loading libraries
Usually this is because you are setting sys.path to point to the libraries, but you have an entry point in your app.yaml that doesn't invoke the code that sets sys.path. One solution is to put the sys.path-setting code in appengine_config.py. On Thu, Jan 17, 2013 at 11:25 AM, Jesús Espejo wrote: > Hello, > > Since 2 days i'm experimenting issues when i load my own libraries > (libraries created by myself and used in mi app): > > Traceback (most recent call last): > File > "/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", > line 196, in Handle > handler = _config_handle.add_wsgi_middleware(self._LoadHandler()) > File > "/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", > line 255, in _LoadHandler > handler = __import__(path[0]) > File > "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/webHandler.py", > line 15, in > from lib.userLib import adminUser > File > "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/userLib/adminUser.py", > line 15, in > import common as CU > File > "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/userLib/common.py", > line 16, in > import normalUser > File > "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/userLib/normalUser.py", > line 16, in > from lib import common, city, images, sport, club > File > "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/club.py", > line 18, in > from lib.userLib import common as CU > ImportError: cannot import name common > > > Sometimes it works, sometimes it doesn't work... What is the reason? For the > same version and without deploying new code, sometimes it works, sometime it > doesn't... > > > Thanks in advance, > > Jesús. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/TmE90_U3WAUJ. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Problems loading libraries
Recursive imports? Missing __init__.py? On Thu, Jan 17, 2013 at 12:31 PM, Jesús Espejo wrote: > Even when i add to my path, it fails :-/. > > I'm showing the path and the libs are included: > > ['/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293', > '/python27_runtime/python27_dist/lib/python27.zip', > '/python27_runtime/python27_dist/lib/python2.7', > '/python27_runtime/python27_dist/lib/python2.7/plat-linux2', > '/python27_runtime/python27_dist/lib/python2.7/lib-tk', > '/python27_runtime/python27_dist/lib/python2.7/lib-old', > '/python27_runtime/python27_dist/lib/python2.7/lib-dynload', > '/python27_runtime/python27_dist/lib/python2.7/site-packages', > '/python27_runtime/python27_lib/versions/1', > '/python27_runtime/python27_lib/versions/third_party/jinja2-2.6', > '/python27_runtime/python27_lib/versions/third_party/markupsafe-0.15', > '/python27_runtime/python27_lib/versions/third_party/setuptools-0.6c11', > '/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2', > '/python27_runtime/python27_lib/versions/third_party/webob-1.1.1', > '/python27_runtime/python27_lib/versions/third_party/yaml-3.10', > '/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293/lib', > '/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293/lib/userLib', > '/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293/lib/apis', > '/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293/pytz'] > > > Why it still continues failing? :-/ > > > El jueves, 17 de enero de 2013 20:25:35 UTC+1, Jesús Espejo escribió: >> >> Hello, >> >> Since 2 days i'm experimenting issues when i load my own libraries >> (libraries created by myself and used in mi app): >> >> Traceback (most recent call last): >> File >> "/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", >> line 196, in Handle >> handler = _config_handle.add_wsgi_middleware(self._LoadHandler()) >> File >> "/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", >> line 255, in _LoadHandler >> handler = __import__(path[0]) >> File >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/webHandler.py", >> line 15, in >> from lib.userLib import adminUser >> File >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/userLib/adminUser.py", >> line 15, in >> import common as CU >> File >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/userLib/common.py", >> line 16, in >> import normalUser >> File >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/userLib/normalUser.py", >> line 16, in >> from lib import common, city, images, sport, club >> File >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/club.py", >> line 18, in >> from lib.userLib import common as CU >> ImportError: cannot import name common >> >> >> Sometimes it works, sometimes it doesn't work... What is the reason? For >> the same version and without deploying new code, sometimes it works, >> sometime it doesn't... >> >> >> Thanks in advance, >> >> Jesús. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/I-Y7zvcZUEoJ. > > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: Problems loading libraries
Either that, or A imports B imports C imports A. I recommend that you try to understand how sys.path and imports work in a smaller example first, before creating something as complex as what you apparently have. :-) On Thu, Jan 17, 2013 at 12:47 PM, Jesús Espejo wrote: > Hello, > > Recursive imports means "libraries inside libraries"? I have lib as library, > and inside it i have another library called "userLib". That's a problem...? > I will try to move to different folders... > > El jueves, 17 de enero de 2013 21:43:07 UTC+1, Guido van Rossum escribió: >> >> Recursive imports? Missing __init__.py? >> >> On Thu, Jan 17, 2013 at 12:31 PM, Jesús Espejo >> wrote: >> > Even when i add to my path, it fails :-/. >> > >> > I'm showing the path and the libs are included: >> > >> > >> > ['/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293', >> > '/python27_runtime/python27_dist/lib/python27.zip', >> > '/python27_runtime/python27_dist/lib/python2.7', >> > '/python27_runtime/python27_dist/lib/python2.7/plat-linux2', >> > '/python27_runtime/python27_dist/lib/python2.7/lib-tk', >> > '/python27_runtime/python27_dist/lib/python2.7/lib-old', >> > '/python27_runtime/python27_dist/lib/python2.7/lib-dynload', >> > '/python27_runtime/python27_dist/lib/python2.7/site-packages', >> > '/python27_runtime/python27_lib/versions/1', >> > '/python27_runtime/python27_lib/versions/third_party/jinja2-2.6', >> > '/python27_runtime/python27_lib/versions/third_party/markupsafe-0.15', >> > '/python27_runtime/python27_lib/versions/third_party/setuptools-0.6c11', >> > '/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2', >> > '/python27_runtime/python27_lib/versions/third_party/webob-1.1.1', >> > '/python27_runtime/python27_lib/versions/third_party/yaml-3.10', >> > >> > '/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293/lib', >> > >> > '/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293/lib/userLib', >> > >> > '/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293/lib/apis', >> > >> > '/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364657294245293293/pytz'] >> > >> > >> > Why it still continues failing? :-/ >> > >> > >> > El jueves, 17 de enero de 2013 20:25:35 UTC+1, Jesús Espejo escribió: >> >> >> >> Hello, >> >> >> >> Since 2 days i'm experimenting issues when i load my own libraries >> >> (libraries created by myself and used in mi app): >> >> >> >> Traceback (most recent call last): >> >> File >> >> >> >> "/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", >> >> line 196, in Handle >> >> handler = _config_handle.add_wsgi_middleware(self._LoadHandler()) >> >> File >> >> >> >> "/python27_runtime/python27_lib/versions/1/google/appengine/runtime/wsgi.py", >> >> line 255, in _LoadHandler >> >> handler = __import__(path[0]) >> >> File >> >> >> >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/webHandler.py", >> >> line 15, in >> >> from lib.userLib import adminUser >> >> File >> >> >> >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/userLib/adminUser.py", >> >> line 15, in >> >> import common as CU >> >> File >> >> >> >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/userLib/common.py", >> >> line 16, in >> >> import normalUser >> >> File >> >> >> >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/userLib/normalUser.py", >> >> line 16, in >> >> from lib import common, city, images, sport, club >> >> File >> >> >> >> "/base/data/home/apps/s~37c2b5db880b7a17476582a0fb597b/1.364655963966448445/lib/club.py", >> >> line 18, in >> >> from lib.userLib import common as CU >> >>
Re: [google-appengine] Re: Cost of updating list property
The only way to get this to perform is to serialize the values into a string. Repeated properties are way too slow when you get to this number of repetitions. (They're good for tags, which need to be indexed, and of which youd expect there to be z few, or dozens, or occasionally hundreds.) --Guido On Fri, Jan 25, 2013 at 11:37 AM, Ryan Chazen wrote: > Depends if those integers are indexed or not. Each index is a write op. > However, an entity can only be 1mb big.. I think 1 million integers would > be more than 1mb as appengine stores natively as strings... > > What you could try is to store the integers in multiple entities (say > 1-1000 integers per entity). That would make adding new integers easy as > you would not need to modify old entities. You could join all the integers > back together if needed with a query. > > > On Friday, January 25, 2013 8:32:48 PM UTC+2, Jiansen He wrote: >> >> Hi, >> >> Say an entity has a list property which has 1 million integers. Now we >> need to update the list property by adding 1 more integer. Does updating >> via put cost 2 million write operations? If so, is there a cheap >> alternative? >> >> With Thanks >> Jiansen >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appengine@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] NDB Parallel Tasklets
On Wed, Jan 30, 2013 at 9:00 AM, Moises Belchin wrote:
> Please take a look at this parallel tasklet code snippet #1.
>
>
> @ndb.tasklet
> def get_data_parallel(e):
> usr, det = yield (e.user.get_async(),
> MyKind.query(ancestor = e.key).fetch_async())
> raise ndb.Return((e, usr, det))
>
>
>
> If e.user is None this raise an Exception.
>
>
> I'm trying this snippet #2. However I still get Exception: "TypeError:
> Expected Future, received : None"
>
>
> @ndb.tasklet
> def get_data_parallel(e):
> usr, det = yield (e.user.get_async() if e.user else None,
> MyKind.query(ancestor = e.key).fetch_async())
> raise ndb.Return((e, usr, det))
>
>
> How can I do something like snippet #2 ? How can I return future(None) or
> future('')
You can factor it out into two yields, one of which is optional. First
create a future for the query that you always want to run:
f = MyKind.query(ancestor = e.key).fetch_async() # No yield!
Then conditionally yield the other async request:
if e.user:
usr = yield from e.user.get_async()
else:
usr = None
Finally yield the future:
det = yield f
The trick is that the query will run when you yield the other operation.
--
--Guido van Rossum (python.org/~guido)
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] NDB Parallel Tasklets
On Thu, Jan 31, 2013 at 2:05 AM, Moises Belchin wrote: > Is there any significant benefit between code#1 and code#2 ?? In docs you > can read code#2 run in parallel. > https://developers.google.com/appengine/docs/python/ndb/async#parallel_queries_yield > >> @ndb.tasklet >> >> def get_cart_plus_offers(acct): >> >> cart, offers = yield get_cart_async(acct), get_offers_async(acct) >> >> raise ndb.Return((cart, offers)) >> >> That yield x, y is important but easy to overlook. If that were two >> separate yield statements, they would happen in series. But yielding a tuple >> of tasklets is a parallel yield: the tasklets can run in parallel and the >> yield waits for all of them to finish and returns the results. (In some >> programming languages, this is known as a barrier.) > > > In appstats there is no significant difference between both options or maybe > I don't see it. It's possible I'm missing something. > > Could someone bring to me more light here ?! > > Thanks in advance and best regards. > > Code#1: > > @ndb.tasklet > def get_data(e): > usr = yield e.user.get_async() > det = yield MyKind.query(ancestor = e.key).fetch_async() > raise ndb.Return((e, usr, det)) > > Code#2: > > @ndb.tasklet > def get_data_parallel(e): > usr, det = yield (e.user.get_async(), MyKind.query(ancestor = > e.key).fetch_async()) > raise ndb.Return((e, usr, det)) Are you looking at Appstats in the dev appserver? It does not give results (in cases like this) that match production. The dev appserver does not really execute RPCs in parallel (which is the same as concurrently, here). I promise you that in production the "yield f1, f2" form runs the tasks represented by futures f1 and f2 concurrently (== in parallel). I should also explain (again) that there is a huge difference between this: f1 = foo_async() f2 = bar_async() yield f1, f2 vs. yield foo_async() yield bar_async() the latter is equivalent to f1 = foo_async() yield f1 f2 = bar_async() yield f2 Because the first future is yielded before the second is even created, nothing runs concurrently (== in parallel) here. However, now compare to this: f1 = foo_async() f2 = bar_async() yield f1 yield f2 This runs both futures in parallel (== concurrently) even though they are yielded separately! The reason is that when you yield *any* future, *all* futures that exist at that point are allowed to run. But futures that haven't been created yet can't run! Hope this helps. It is important to "get" this. (Also that no future runs until you yield something. Futures are buffered in the app's memory until a yield forces all buffered futures out to the servers.) -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [google-appengine] Large number of sites all over the world request an image twice within a second or so
uot;GET >> > /fish/goldfish/%2Bimg/**goldfish.jpg HTTP/1.1" 404 1012 - >> "Mozilla/4.0" >> > "www.fishandlily.com.au" ms=175 cpu_ms=42 cpm_usd=0.000113 >> > >> > The total number of IP's is too large to for DOS blacklist. >> > >> > Any one got any ideas what is going on. >> > >> > Cheers >> > >> > Tim >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Google App Engine" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an >> > email to google-appengi...@**googlegroups.com. >> > To post to this group, send email to google-a...@googlegroups.**com. >> > Visit this group at http://groups.google.com/** >> group/google-appengine?hl=en<http://groups.google.com/group/google-appengine?hl=en>. >> >> > For more options, visit >> > https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>. >> >> > >> > >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to google-appengine+unsubscr...@googlegroups.com. > To post to this group, send email to google-appengine@googlegroups.com. > Visit this group at http://groups.google.com/group/google-appengine?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: Zip Importer High CPU every request
One question -- are you defining a main() function? If not, your entire app may be reloaded each time. Try putting logging statements (or prints to sys.stderr) at the top-level of your module, and check in the logs for these -- they should only be logged for the first request. If they are logged for each request, then somehow your main is being reloaded each time. On Jan 7, 4:39 pm, Anthony wrote: > It looks like the info in the logs could be down to the whole page > processing, but very simple pages (no db) consistently taking 500ms+ > Ill give it a try without the zipimport. > > I'm just using the loader cache from the cookbook (removed the > bytecode cache), jinja works great, although I did need to force > caching off in dev or the internal caches do not update after template > changes... > > import config > import os > import sys > from google.appengine.api import memcache > > sys.path.insert(0, 'lib/jinja2.zip') > from jinja2 import Environment, FunctionLoader , TemplateNotFound > > def jinja2_template_loader(templatename): > templatepath = os.path.abspath(os.curdir+'/templates/'+templatename) > template = memcache.get(templatepath+config.VERSION) > if template is None: > try: > template = file(templatepath).read() > if config.LIVE: > > memcache.set(templatepath+config.VERSION,template) > except: > template = None > return template > > if config.LIVE: > jinja2_environment = Environment(loader = FunctionLoader > (jinja2_template_loader),cache_size=50) > else: > jinja2_environment = Environment(loader = FunctionLoader > (jinja2_template_loader),cache_size=0) > > def render(template_name,context): > template = jinja2_environment.get_template(template_name) > context["stats"]="LIVE:" +str(config.LIVE) + " DEBUG:" +str > (config.DEBUG) > return template.render(context) > > On Jan 7, 7:39 pm, "Rodrigo Moraes" wrote: > > > On Wed, Jan 7, 2009 at 2:16 PM, Anthony wrote: > > > Is it normal for the zipimporter to take 500-1000ms every request - > > > should it be caching internally? > > > > I'm importing like so: > > > > # templates.py > > > sys.path.insert(0, 'lib/jinja2.zip') > > > from jinja2 import Environment, FunctionLoader , TemplateNotFound , > > > MemcachedBytecodeCache > > > > Will it still be caching if im not importing the zip dircetly in the > > > main.py? > > > Not an answer to your question (which I'd like to know :), just a > > warning: you won't be able to use MemcachedBytecodeCache from Jinja2, > > because bytecode usage is not allowed in App Engine. > > > The example using MemcachedBytecodeCache in the App Engine Cookbook > > only works in the dev server, not in production. :-/ > > > Apart from this, Jinja2 implementation is very smooth. :) > > > -- rodrigo --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: files, directories & symlinks in root directory
See http://code.google.com/appengine/docs/python/tools/configuration.html#Static_File_Handlers "For efficiency, App Engine stores and serves static files separately from application files. Static files are not available in the application's file system. If you have data files that need to be read by the application code, the data files must be application files, and must not be matched by a static file pattern." What this is saying is that any file that is matched by a "static" rule in your app.yaml is only available as a static URL, and cannot be read directly by your app. The symlink work-around you discovered this works because the app uploader treats symlinks as if they were regular files, so it's as if a copy of form.html exists in your root dir. --Guido On Feb 10, 8:08 pm, Jason Dusek wrote: > I have encountered a curious difficulty. Here's how my project > directory is set up: > > ./ > app.yaml > css/ > clear.css > green.css > form.html -> html/form.html > form.py > html/ > blank.html > form.html > index.yaml > js/ > jquery-1.3.1.min.js > > Why do I have that symlink there? It's because I can't access > the HTML directory (or other directories) when my app is > deployed on Google apps. I can access them fine when I working > locally; I also know they are being uploaded, because I can > visit, for example: > > http://.appspot.com/html/form.html > > When I set up my Python program to just respond with a > directoy listing, it shows me this on my laptop: > > ['app.yaml', 'css', 'form.html', 'form.py', 'html', 'index.yaml', > 'js'] > > while when I deploy the app, I get: > > ['form.py', 'form.html'] > > Now it is clear why `app.yaml` went away -- it does not belong > in the application -- but why don't I see `js`, `css` and > `html`? This is especially strange because they are referenced > in the `app.yaml` file and, as mentioned previously in this > message, they are reachable through the URL handlers I set up. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
Re: [google-appengine] modifying an existing datamodel in the future.
Hm, actually, he's talking about Java, where the situation is completely different. AFAIK there is *another* SQL-like language (this one much closer to "real" SQL) that is part of a standard (not App Engine specific) way of accessing databases. IIUC this is part of JDO -- search for Datanucleus and SQL and you'll find it. JDO and Datanucleus are not App Engine specific (and not written by us). However in App Engine certain queries (e.g. Joins) are not supported. But I don't think we have the option of not calling it SQL. Still, thanks for forwarding this -- I do understand that GQL is controversial (several users have already pleaded to keep it). --Guido On Mon, Jan 17, 2011 at 21:01, Robert Kluin wrote: > Guido, > Just wanted to point this post out, it is a good example of what > several people recently mentioned about GQL and how it leads to > confusion with SQL. > > I think this line summarizes some people's confusion well: > "use them with googles app engine sql language." > > > I don't have any good suggestions for a solution, since I know many > people really like *G*QL. Other than maybe changing the name to > something not SQL sounding, maybe... EFL -- Entity Finding Language > ... :) > > > > Robert > > > > > > On Mon, Jan 17, 2011 at 10:11, Steel City Phantom wrote: >> im writing a new app. currently its a LAMP app, heavy on AJAX, but once >> this prototype is done and running i want to rewrite it as an enterprise >> level application. im looking at GAE as a back end (more because ive never >> done it before and it looks cool than anything else, free hosting doesn't >> hurt either) but im struggling with the data store stuff. the biggest >> problem i have so far is changing the data store once its deployed. here is >> what i understand so far, tell me where im wrong cuz im sure i am >> 1 - create your entity objects (this is java) >> 2 - when deployed, GAE will create the data store based on the entity >> objects in your WAR file. >> 3 - use them with googles app engine sql language. >> now, if i want to change the model >> update your entity objects with your new fields or whatever and deploy. now >> GAE should maintain the existing data and update the objects. >> or am i completely off base in thinking like a relational database system. >> is it more as a base relational system that stores blobs and i can store >> whatever i want in them? >> second, >> what if i have a drastic change to the data model that requires some >> translation sql to implement? is there some kind of screen where i can >> query and update the existing data model manually and translate the data >> into the new model or do i have to do it in the application code. >> >> -- >> You want it fast, cheap, or right. Pick two!! >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to google-appengine@googlegroups.com. >> To unsubscribe from this group, send email to >> google-appengine+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> > -- --Guido van Rossum (python.org/~guido) -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
