[google-appengine] Re: Instances still getting killed and causing user 500's
I'm also getting this problem. On Wednesday, February 27, 2013 7:50:47 AM UTC-8, Kinesh Patel wrote: > > On Python 2.7 paid app. Setting an idle instance seems to mitigate this, > but every time a new dynamic instance is spawned or hit i get this problem. > Definitely not normal behavior. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[google-appengine] Re: GAE and AWS
This isn't the right place to post (try Amazon's forums). Nonetheless, I recommend boto (http://code.google.com/p/boto/) for a nice library for working with amazon's web services. A big caveat is that you will have to substitute google's urlfetch where appropriate. On Aug 28, 6:09 pm, Davide Rognoni <[EMAIL PROTECTED]> wrote: > Amazon Web Serviceshttp://aws.amazon.com/ > > The best way is the cooperation, but I don't see > Pythonhttp://docs.amazonwebservices.com/AmazonSimpleDB/2007-11-07/GettingSt... > > Java > C# > Perl > PHP > VB.NET > > and my Python :-( --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: ANNOUNCE: App Engine Console 1.0 beta 1
I've only tested the shell, and it looks quite nice :-) One suggestion I'd make is to set the the "console_area" div height to a percentage so that users with small screens don't have to scroll to input commands. Have you thought about implementing auto-completion? -Chris On Sep 5, 11:48 am, Jason Smith <[EMAIL PROTECTED]> wrote: > I am pleased to announce the initial public version of App Engine > Console, a free software tool to assist with App Engine development. > Please see the following sites to get started: > > * Online demo:http://con.appspot.com/ > * Project page:http://www.proven-corporation.com/software/app-engine-console/ > > App Engine Console is a traditional interactive Python session running > in the server, which you use from a web browser. If you are familiar > with the Ruby on Rails or Django command-line consoles, then App > Engine Console is the same thing, except it is web based (AJAX). > > Please see the site for details, but here are the main features: > * Just like running Python interactively, but with syntax > highlighting > * Works from the SDK or in production (great for advanced datastore > management) > * Exceptions, modules, and types link to their own documentation on > python.org > * Quick access to the datastore, application dashboard, and Google > Analytics > * When in production, optionally returns HTTP 404 to non-authorized > users (”cloaked mode”) > > I have tested with Firefox and Internet Explorer (6 and 7). I intend > to support more browsers, but at this point I would like to get the > software out there and see what kind of feedback I get. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Online development environment for GAE
I've been thinking about building a web based python IDE. One of the problems with appengine version control is that it's too simple. No branches, merges, etc, means it lacks collaboration features. Unfortunately, the python-svn library requires the svn binary, which won't run on appengine. Working with Git faces the same problems. Enabling good version control would require implementing the SVN/Git HTTP protocols or running part of the software on a non-appengine server. An exciting feature would be live, collaborative coding. Live, meaning code which is changed is instantly reflected in the application, and collaborative in that you could work together with other programmers in real time. I don't think either feature would be very difficult to implement. Live coding is already working in the dev-appserver, and there exists a google library for collaborative editing of text files (http://code.google.com/p/google- mobwrite/). Anyone interested in working on this project together? On Sep 5, 10:35 am, windchess <[EMAIL PROTECTED]> wrote: > Has anybody heard if Google is planning on creating an online > development environment for GAE ? > > It would make it really easy to get started using GAE without having > to download and install anything on your computer. > > Several benefits I see: > - no need to install any software locally > - no need to manage versions/releases of GAE > - much easier to get started with it > - no need to deploy from local environment to Google > - much easier to work from any computer > > GAE could also manage the version of the code for us by default... --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: ANNOUNCE: App Engine Console 1.0 beta 1
On Sep 8, 2:40 am, Jason Smith <[EMAIL PROTECTED]> wrote: > Thanks, Chris. > > For now, the interface is optimized for 1024x768 resolution or larger, > which represents 99.5% of the visitors to the site. To me, growing > the console to fit larger screens is a good goal too--but why not fix > both bugs together? I will support a full-height console for the next > release, for all resolutions. > The reason I noticed this is because i'm using an EEE Pc with a puny 1024x600 res :) > I have considered auto-completion as well. I had an unsuccessful > trial with IPython on the back-end, but it was not simple to integrate > in such an un-terminal-like environment. So my plan B was to make it > as much like the plain-Jane Python console as possible. However, > autocompletion should be quite straightforward to implement; a simple > approach would be a new web service which just returns a clean list of > your past input (and/or output). Then the Javascript could take care > of the rest client-side. Another possibility is to go with something > like YUI, which has its own autocompletion features, not to mention > consistent behavior across browsers. For simple auto-completion, I think these steps might work: 1. Client types "foo" in console 2. Server looks up the name of the object in the current namespace and does a "dir(foo)" to find out the attributes of the object. Perhaps it also does some sorting to show the underscored attributes last. 3. Client receives suggestion list and picks the variable "bar" which happens to be a method 4. Server looks up the docstring "foo.bar.__doc__" of the method and sends it to the client. The server can also introspect on the method's variables using "foo.bar.func_code.co_varnames" or the inspect module's getargspec --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: ANNOUNCE: App Engine Console 1.0 beta 1
> Yes I think this is a decent approach. However I think something is > missing between #2 and #3. #3 sounds to me like the *user* picks the > variable bar, which implies that somehow he saw a list of attributes > and somehow chose one. So there needs to be some sort of client-side > presentation to show the user all the choices. I see two options, Vim > style and Bash/IPython style. Vim style lets the tab key cycle > through valid options in alphabetical order. Bash/IPython style > displays all the choices in a listing, and the user continues typing > from where he left off. Once he has identified a unique name from the > list, the tab key finishes typing it out for him. > > I am definitely leaning toward Bash/IPython style, however the only > problem is that if you copy and paste the console session, it will not > look like a pure Python session anymore, but more like an IPython > session. I guess that's not the end of the world, though. I was thinking of a pop-up list of suggestions above the variable name. The python IDE which I use (Stani's Python Editor) supports this. It's also open source, so you could have a peek at how they implement this. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Announcement: Pycmds, a web-based command line
http://www.pycmds.org Features: Auto-suggest for commands and inputs Auto-complete Simple API There are still many kinks to work out with the client side code. Suggestions & comments are appreciated :) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: datastore design for hierarchical data.
I agree with this approach. It works well with tree data containing fewer than 1000 nodes. It's also the amount of entities that appengine can fetch in one query. Here's some example code which I'm using for a forum, it may help you get started: http://gist.github.com/370861 It's running at: http://www.caterpi.com/forum On Apr 18, 8:03 am, Tim Hoffman wrote: > Here is the approach I would probably take unless you have many > hundreds of child tasks. > > I would keep a list of keys of all children of each project/task. > (You can then do a db.get and get all immediate children with single > call) > and each child has a reference to it's parent. In addition in each > item keep a list of all parents keys back to the root. That way you > can search for all children > in part of the tree/subtree > > (This approach is especially useful if you have multiple child types) > as back ref sets > are kind specific. > > Keep a running count of complete children at each level, rather than > trying to calculate the current state in a single big function, (keep > these subsidiary values up to date through tasks) > > T > > On Apr 18, 8:29 pm, gops wrote: > > > > > hi , i am designing a simple project based to do list. the idea is to > > define tasks under project ( no workflow - just "task is completed" or > > not is required. ) in a hirarchial way. i.e. each task has multiple > > task and that task may have other multiple task. a project can be said > > to be completed if all task under that project are completed. , i > > tought of using refrenceproeperty to create hirarchy , but could not > > figure out easy way ( which do not take more than 30 seconds to find > > all the children of a project and check weather it is completed or > > not ) . to detect if project is complete or not. how to design > > database for such job ? and also , if i need to copy the project in > > order to define another project , how to copy hierarchical data ? > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to google-appeng...@googlegroups.com. > To unsubscribe from this group, send email to > google-appengine+unsubscr...@googlegroups.com. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: Announcing EnStore
Your team has done an amazing job with the design and layout! Best of luck on your launch, Chris On Sep 24, 4:44 pm, Koen Bok wrote: > Hi everyone. I'm pleased to announce our ecommerce platform built on > appengine:http://www.enstore.com > > Apart from some DeadlineExceeded errors, the launch went pretty > smooth. Currently, a ton of Checkout users are syncing their products > to build a webstore. Thanks appengine team! > > Koen --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: vote for native Django support
Haha, very cute! On Oct 14, 6:35 pm, "left.pudd...@gmail.com" wrote: > I created logo to support it. Native porting is our dream. > > http://helloleftpudding.appspot.com/image/real_django_porting.jpg --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Prettier values for reference keys?
I wrote a KeyNameReferenceProperty awhile back, maybe this will help: http://gist.github.com/109193 On Oct 12, 12:45 am, Tonny wrote: > Hi > > Can someone help with this consideration: > > When we're creating a ReferenceProperty on datastore objects the value > stored in the model is the key > for the referenced object (stating the obvious here, but please stay > with me), but when the reference object > has a key_name, which makes up a pretty name (understandable from the > datastore viewer), it would be nice to use that key name instead. Is > there an easy way to use that key_name instead? > > I could imaging subclassing ReferenceProperty, but looking at the API, > I can't seem to find someplace to hook > in. I could write my own ReferenceProperty from scratch, subclassing > Property, but I would also like a simple > solution where I didn't reinvent all ready existing functionality. > > Regards > Tonny --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Modeling Hierarchical Data
I'm wondering how other people have been modeling hierarchical information such as nested comments. My first thought was to do something like this: class Comment(db.Model): reply_to = db.SelfReferenceProperty(collection_name="replies") author = db.UserProperty() content = db.TextProperty() However, this isn't scalable, as a query is made for every comment in the thread. Any ideas? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Modeling Hierarchical Data
Thanks for the heads up on that interesting method. Just wondering, from your comment in the other thread: "Or, stick with the lexical path but use the full byte rather than decimal to store the number.. then just 2 bytes give you a max of 64k comments per depth and 200+ deep. " The sort order for StringProperty is in Unicode, so wouldn't you have to use an something like UTF-8? Some pros of this method: - Fast query for children - Possible to reconstruct hierarchy Cons: - Limited depth - Need to do query for siblings before setting the path - Somewhat complex. Need to increment Unicode character On Oct 31, 3:34 am, Anthony <[EMAIL PROTECTED]> wrote: > Store all the parent ids as a hierarchy in a string... > > h="id1/id2/" > h="id1/id2/id3" > h="id1/id2/id3/id4" > h="id5/" > h="id5/id6" > > You can then filter WHERE h > "id1/" and h < "id5/" to get all > children of id1. > > Or you can use entity groups & ancestors. > > More details > here...http://groups.google.com/group/google-appengine/browse_thread/thread/... > > On Oct 31, 8:31 am, Chris Tan <[EMAIL PROTECTED]> wrote: > > > I'm wondering how other people have been modeling hierarchical > > information > > such as nested comments. > > > My first thought was to do something like this: > > > class Comment(db.Model): > > reply_to = db.SelfReferenceProperty(collection_name="replies") > > author = db.UserProperty() > > content = db.TextProperty() > > > However, this isn't scalable, as a query is made for every > > comment in the thread. > > > Any ideas? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Modeling Hierarchical Data
Hi Steve, I was thinking along the same lines. Having the root comment or thread hold a cache of the whole tree, which would be updated whenever an entity is updated, added or removed seems like the most efficient way for large hierarchies. You could have your cached tree represented using nested dictionary objects on your root object using a PickleProperty (see: http://groups.google.com/group/google-appengine/msg/8433525107a8bb92) to be converted to XML as required. The comment class could have a path property (e.g. "id1/id2/id3") which could be quickly traversed to retrieve or update child comments. Since it doesn't need to be indexed, it can be a TextProperty() which isn't limited to 500 bytes. CRUD operations could be overridden in your Model class: class Comment(db.Model): path = TextProperty() def put(self): # update root comment super(Comment, self).save() def delete(self): # update root comment super(Comment, self).delete() Pros: - Reads only need to fetch 1 (root tree) or 2 entities (look-up child path & return sub-tree from root) - Not limited by depth Cons: - Writes update 2 entities. - Pickling/Unpickling could be resource intensive On Oct 31, 2:11 pm, "Steve Schwarz" <[EMAIL PROTECTED]> wrote: > On Fri, Oct 31, 2008 at 3:18 PM, Calvin Spealman <[EMAIL PROTECTED]>wrote: > > > It seems to me that the question at hand isn't hierarchial entities, per > > se. We have quite a few options for representing them and any will do, > > truthfully. > > > It is the operations on them which pose issues. Notably, how do we make our > > queries for decendants cost-effective? > > > I say use whatever representation feels comfortablee to you. Other issues > > you may deal with are a matter of indexing and other operations both > > important and separate from the exact representation. If the need is "give > > me all of this node's decendants" then we may create simple index entries > > mapping a node to each of its decendants and mmay be in a group without > > impacting the nodes themselves or other nodes' decendant indexes. > > Calvin, > I very new to GAE but that does seem to be the issue Chris was posing. While > the "natural" representation is a tree structure, the cost of getting each > comment might exceed the compute/selects allowed by GAE w/in a request. > Imagine if these were the comments for some popular Slashdot article. At > what point do you optimize for read speed? I'd guess the use case for > comments is "read many write few". It could be that the best model is to > actually update a single comment object with new comments (it would store > the hierarchy) so that the read case involves querying for a single large > object. > > An alternative is to store each comment in a hierarchy as described in > Chris' email and have an external periodic job hit the server and update a > single comment instance with the hierarchy when it detects that new comments > have been added (a kind of external mapreduce). Or a combination that > updates the "master" comment with the new comment when it is added. If you > want to do moderation the moderation step could be the one that appends the > new comment to the master comment. > > I'm thrashing on a similar problem. I want to serve a relatively large XML > file containing a couple thousand elements. It will have very few updates > and many reads. I'd like to be able to perform CRUD operations on the > elements and then have the XML file be updated when any of those elements > change. I don't want to prematurely optimize, but then again don't want to > have a "non-starter" either. > > From reading this list and the docs it seems there are times when > traditional DB backed designs have to be modified to work well in the GAE > environment. I'm also looking for direction/patterns/best practices. > > Best Regards, > Steve --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Modeling Hierarchical Data
@Anthony Very nice method that seems to work better than storing the hierarchy in a string. I'll have to look into that. @Marzia Thanks for the advice =) It doesn't suit my particular use case, since my layout displays nested comments that are ~10-20 deep, but I'm sure many people will find the lazy-loading style useful. @Steve I've been experimenting with pickling a highly nested tree. Seems like I ran into a problem with the recursion limit in the pickle module. The way I solved this was to flatten the tree, however, you could also use a patched pickle which uses generators instead of recursion. It's posted at: http://gist.github.com/22217 On Nov 1, 7:31 am, "Steve Schwarz" <[EMAIL PROTECTED]> wrote: > On Fri, Oct 31, 2008 at 5:41 PM, Chris Tan <[EMAIL PROTECTED]> wrote: > > > Hi Steve, > > > I was thinking along the same lines. Having the root comment or > > thread hold a > > cache of the whole tree, which would be updated whenever an entity is > > updated, added > > or removed seems like the most efficient way for large hierarchies. > > > You could have your cached tree represented using nested dictionary > > objects on your root > > object using a PickleProperty (see: > >http://groups.google.com/group/google-appengine/msg/8433525107a8bb92) > > to be converted to XML as required. > > > The comment class could have a path property (e.g. "id1/id2/id3") > > which could be > > quickly traversed to retrieve or update child comments. Since it > > doesn't need to be indexed, > > it can be a TextProperty() which isn't limited to 500 bytes. > > > CRUD operations could be overridden in your Model class: > > > class Comment(db.Model): > > path = TextProperty() > > def put(self): > > # update root comment > > super(Comment, self).save() > > > def delete(self): > > # update root comment > > super(Comment, self).delete() > > > Pros: > > - Reads only need to fetch 1 (root tree) or 2 entities (look-up child > > path & return sub-tree from root) > > - Not limited by depth > > > Cons: > > - Writes update 2 entities. > > - Pickling/Unpickling could be resource intensive > > Chris, > Now that I've read Marzia's comments and rewatched the video the saving > grace for a comment system where you get "too many" comments to grab in one > go you could switch to paging. Whatever the GAE limiting factor is once you > near it you could introduce paging to keep each request under the limit. If > that was 200 or 1000 comments would anyone really want to scroll through > them all on a single page? > > On the other hand if you are providing a data feed or a big XML doc like I'm > contemplating then getting only a portion of the data wouldn't be useable. > But now that I think of it there is no reason the tree couldn't be > partitioned and at the cost of some additional gets (which could be cached) > this becomes as sharded assembly of the larger XML document. Just like > comments could be put in entity groups based on user, I could split the XML > doc based on the values of a discriminant attribute and work with smaller > entity groups. The assembly of those entities into the final document would > involve a relatively small number of gets (a couple dozen). I'll have to > look into pickling vs. storing XML fragments from a speed/space tradeoff > perspective. > > Thanks for your assistance, > Steve --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: URLlib?
A few projects have monkey-patched httplib which allows you to use urllib & urllib2. See appengine-patch & appengine-monkey for more info On Dec 13, 7:50 pm, "moham...@ahft.org" wrote: > is it posable to use it(urllib)? It will not let the opener init. so > how can I run this? vital to my app! ugh, looking for work arounds, > but heading to bed now, so perhaps someone else knows something? Also > Google App admins, shouldn't that just work? why is it not working? > > error 1 > Traceback (most recent call last): > File "/base/python_lib/versions/1/google/appengine/ext/webapp/ > __init__.py", line 501, in __call__ > handler.post(*groups) > File "/base/data/home/apps/youtubedownpsp/1.329967885382643345/ > main.py", line 22, in post > f = urllib.urlopen(url) > File "/base/python_dist/lib/python2.5/urllib.py", line 77, in > urlopen > opener = FancyURLopener() > File "/base/python_dist/lib/python2.5/urllib.py", line 609, in > __init__ > URLopener.__init__(self, *args, **kwargs) > File "/base/python_dist/lib/python2.5/urllib.py", line 123, in > __init__ > self.__unlink = os.unlink # See cleanup() > AttributeError: 'module' object has no attribute 'unlink' > error 2 > Exception exceptions.AttributeError: "FancyURLopener instance has no > attribute 'tempcache'" in > ignored --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Key Vs Reference
Reference property fetches by key. Back-references run a query, and are slower than fetching a list of keys. Both of these are handled transparently using python properties: http://www.python.org/download/releases/2.2.3/descrintro/#property -Chris On Dec 20, 11:41 pm, Guy Rutenberg wrote: > Hi, > > Say I've a key for an item and a reference to it, what would be > faster: getting it by key or via the reference? > > Also, would fetching by a list of keys be slower than fetching by > using a back-reference? > > Thanks, > > Guy --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: Converting "François Pétill ant" to UTF-8
I've been using the code below for normalization. Any characters
without
ascii equivalents are stripped:
import unicodedata
nfkd = unicodedata.normalize('NFKD', data)
normalized = nfkd.encode('ascii', 'ignore').lower()
It seems to work well so far for prefix suggest, except when the
datastore
query contains spaces it will fail. Ideas anyone?
On Jan 13, 8:25 am, Tom wrote:
> I need some suggestions/guidance on how to handle strings that contain
> characters like "François Pétillant".
>
> I have several problems.
>
> First, I do want to store these names in their original encoding
> (which I think I can do in a db.Text object).
>
> I also want to be able to search their names with or without the
> special characters. (eg. "François" and "Francois" would both in the
> SearchableModel's tags) Any utility to convert sensibly?
>
> Finally, I want to be able to send back the name in utf-8 in a format
> that can be converted back to the original on the user side. (In this
> specific case the wine name is being sent back to my android phone
> app.)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Generating cryptographically strong random numbers
According to the docs, "On a UNIX-like system this will query /dev/ urandom": http://docs.python.org/library/os.html These libraries may also be of interest to you: http://docs.python.org/library/crypto.html http://docs.python.org/library/random.html http://docs.python.org/library/uuid.html Personally, I'm using random.random() for CSRF tokens and it works great. I'm curious, what kind of application are you developing that requires this level of security? On Jan 16, 1:16 am, "Alexander Konovalenko" wrote: > Web applications that implement user accounts need to generate > unpredictable values for session cookies and anti-CSRF tokens. So > several questions about App Engine arise: > > 1. How is os.urandom() implemented in production? That is, what > entropy sources does it use? What RNG algorithm? > > 2. Is there a supported way to obtain some high-entropy bits (like > /dev/random in Linux)? > > 3. Are there any features that could help us generate > cryptographically strong random numbers? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: How to get url from fetchurl?
You can follow redirects manually like so:
response = urlfetch.fetch(url, allow_truncated=True,
follow_redirects=False)
if 'Location' in response.headers:
url = response.headers.get('Location')
response = urlfetch.fetch(url, allow_truncated=True)
You could also loop this to follow more than one redirect.
On Jan 17, 10:18 am, ehmo wrote:
> I want to know actual url, where i'm via fetchurl. Something like
> geturl in urllib2.
>
> for example if i've follow_redirects=True, then i want to know which
> url is the ending one.
>
> any idea?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Generating cryptographically strong random numbers
On Jan 18, 5:54 am, "Alexander Konovalenko" wrote:
> Thanks for you reply, Chris. I thought there wouldn't be any which is
> kind of sad.
>
No problem. I thought it was an interesting question :)
> On Sun, Jan 18, 2009, Chris Tan wrote:
>
> > According to the docs, "On a UNIX-like system this will query /dev/
> > urandom":
> >http://docs.python.org/library/os.html
>
> Sure. The question is what kernel do they run there and whether it is
> configured carefully to avoid RNG-related weaknesses. Is it Linux
> 2.6.x? Does it have enough entropy just after reboot? Can it under any
> circumstances revert the RNG to some previous state (e.g., after
> restoring from a backup)?
>
> Or was os.urandom() manually altered for App Engine to use some
> Google-specific random number generator?
>
That's a question only the Google employees can answer.
> > These libraries may also be of interest to you:
> >http://docs.python.org/library/crypto.html
>
> I'm not sure I get how HMAC and the hash functions are related to
> this. Do you mean some specific way to generate random numbers with
> them?
>
My apologies, I hadn't looked over it carefully.
> >http://docs.python.org/library/random.html
>
> (See below.)
>
> >http://docs.python.org/library/uuid.html
>
> The docs don't specify how exactly does uuid.uuid4() generate a random
> UUID. The source code tries to call some C library function that I'm
> not sure where to look for (and if that function does not exist,
> uuid4() just uses os.urandom()). In that light, sticking to
> os.urandom() seems safer.
>
The docs mention that the functions follow the RFC 4122 (http://
rfc.net/rfc4122.html) guidelines for generating UUIDs. The site is
down right now, but you might be able to find a google cache of it.
> > Personally, I'm using random.random() for CSRF tokens and it works
> > great. I'm curious, what kind of application are you developing that
> > requires this level of security?
>
> How do you know when it stops working great and starts being exploited?
>
> random.random() uses Mersenne Twister which is not cryptographically
> strong. If you are just using the value returned by random.random() as
> your secret token or session cookie, it should be possible for an
> attacker to predict future secret tokens by observing a sufficiently
> large sample of generated random values. According to what I read, a
> sample on the order of 1000 values may be enough. [1]
>
> Switching to random.SystemRandom which uses os.urandom() would be
> wiser, but that method still depends on the security of App Engine's
> os.urandom().
>
> -- Alexander
>
> [1] See section 1.6 ("Limitation and hints for use") of the original
> paper by M. Matsumoto and T. Nishimura "Mersenne Twister: A
> 623-dimensionally equidistributed uniform pseudorandom number
> generator".http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/ARTICLES/mt.pdf
>
> For an estimate of the required sample size, see "A Cryptographic
> Compendium" by J. Savard, section "The Mersenne Twister"
> <http://www.quadibloc.com/crypto/co4814.htm>, subsection called
> "Applications to Cryptography". He writes:
>
> "Despite the fact that the Mersenne Twister is an extremely good
> pseudo-random number generator, it is not cryptographically secure by
> itself for a very simple reason. It is possible to determine all
> future states of the generator from the state the generator has at any
> given time, and either 624 32-bit outputs, or 19,937 one-bit outputs
> are sufficient to provide that state."
It's actually even weaker than that. A man in the middle attacker
(e.g. a wifi provider) wouldn't even need to predict the next token;
sniffing the current token and inserting a script tag into the next
page the user requests would work perfectly (I'm using the cookie
method).
That being said, the level of security being employed should be
dependent on the application. I feel that my app is "secure enough",
and in the unlikely situation that someone is knowledgeable and
motivated enough to compromise it, I still have backups.
Rolling your own security can be a big time sink when frameworks such
as Django do it well by default. If you are starting to worry about
your crypto being broken, maybe appengine isn't the right platform.
Google employees are only human, and who knows when one of them will
be pulled to the "dark side" =P
Thanks for the reply Alex, it was very informative.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: sanitizing user submitted HTML
Check out:
http://feedparser.org/docs/html-sanitization.html
On Jan 21, 2:47 pm, Dave wrote:
> There must be an easy answer for this problem and I almost feel dumb
> for asking BUT I can't figure it out and have spent too much time
> trying. The scenerio is a comment/blog situation. I am using tinyMCE
> which is creating 'trustable' html. I can display this with django by
> using {{field|safe}}... all is good.
>
> The problem is some bozo will have their way with the textarea by
> turning of their javascript. So I'm trying to figure out best way to
> sanitize the data. The normal escaping of data won't work because it
> clobbers the 'good' html from tinyMCE. Anyway would be good to
> sanitize even the tinyMCE generated html.
>
> I've been looking at using html5 lib/parser but can't seem to get it
> to work. I've even gone through creating a replace method to escape
> everything and then put back the 'good' tags. However, that seems like
> a round-about way to go and get's really nasty when considering img,
> span, etc. tags tinyMCE creates so nicely. Surely many have come
> across this and there an easy answer.
>
> All suggestions and recommendations are greatly appreciated.
>
> thx,
>
> Dave
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: sanitizing user submitted HTML
Hi Dave,
Html5lib looks like a well maintained and active project.
The stack overflow clone white-lists a subset of the default safe
elements (e.g. no button elements) which looks alright to me. Of
course, I'm no expert at this, so don't quote me on that :)
Thanks for sharing,
Chris
On Jan 21, 4:36 pm, Dave wrote:
> Thanks Chris and Alexander,
>
> I took a look at both... from the links I also
> foundhttp://code.google.com/p/soclone/source/browse/trunk/soclone/utils/ht...
> which uses html5lib. It puts a wrapper on html5lib and helped me
> figure out how to make it work.
>
> What is wicked cool is that what appeared to be a nightmare seems to
> work just great. For others attempting same thing do this:
> 1- get & install html5lib. Note: phthon manage.py install failed for
> me so i just copied it to my project folder.
> 2- get the code from link above and save it file in your project (i.e.
> htmlsanitize.py)
> 3- I run the code as a clean in my forms(i.e. def clean_comment) such
> as below:
>
> def clean_comment(self):
> import htmlsanitize
> data =
> htmlsanitize.sanitize_html(self.cleaned_data['comment'])
> return data
>
> So far so good for me.
>
> Would love to hear 'thumbs up' or 'thumbs down' if this is a good
> approach.
>
> thx again
>
> Dave
>
> Chris Tan wrote:
> > Check out:
> >http://feedparser.org/docs/html-sanitization.html
>
> > On Jan 21, 2:47 pm, Dave wrote:
> > > There must be an easy answer for this problem and I almost feel dumb
> > > for asking BUT I can't figure it out and have spent too much time
> > > trying. The scenerio is a comment/blog situation. I am using tinyMCE
> > > which is creating 'trustable' html. I can display this with django by
> > > using {{field|safe}}... all is good.
>
> > > The problem is some bozo will have their way with the textarea by
> > > turning of their javascript. So I'm trying to figure out best way to
> > > sanitize the data. The normal escaping of data won't work because it
> > > clobbers the 'good' html from tinyMCE. Anyway would be good to
> > > sanitize even the tinyMCE generated html.
>
> > > I've been looking at using html5 lib/parser but can't seem to get it
> > > to work. I've even gone through creating a replace method to escape
> > > everything and then put back the 'good' tags. However, that seems like
> > > a round-about way to go and get's really nasty when considering img,
> > > span, etc. tags tinyMCE creates so nicely. Surely many have come
> > > across this and there an easy answer.
>
> > > All suggestions and recommendations are greatly appreciated.
>
> > > thx,
>
> > > Dave
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Re: Complex tagging of items
Since you only have a few hundred items, the easiest approach may be to filter the results in memory after performing a preliminary datastore query. I would store the tag names in a StringListProperty () on your model. Hope this helps, Chris On Jan 23, 5:17 pm, George Sudarkoff wrote: > I have a bit of a problem coming up with an efficient data model/algo > for a project I am working on: > > I have a few hundred items, each tagged with zero or more tags. I need > to be able to fetch items that, for example, are tagged with tag1 AND > (tag2 OR tag3) AND NOT tag4. > > Any help would be greatly appreciated! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: memcache.add => RuntimeError: maximum recursion depth exceeded
If you have deeply nested data, you can work around it by:
1. flattening your data
2. using a patched pickle module which does not have the recursion
limit:
http://gist.github.com/22217
To be honest, I don't see why the above code would be causing this.
If you still have problems with this, it may be a good idea to post
the source for the entire module.
Chris
On Feb 1, 3:48 pm, MatthewRudy wrote:
> I've got a recursion error in both my local code, and when deployed.
> But it doesn't make much sense.
>
> At first I thought it was to do with mixing unicode strings in
> dictionaries.
> But I've ironed that out.
>
> Hopefully this is a simple mistake?
>
> @@@ python
>
> splashes = [
> {
> 'url': splash.url,
> 'title': splash.title(),
> 'thumbnail_url': splash.thumbnail_url()
> }
> for splash in record.url_records()
> ]
> logging.info("setting values: %s" % splashes)
> memcache.add(cache_key, splashes, 60)
>
> @@@
>
> and the log output when it breaks;
>
> @@@ log
>
> INFO 2009-02-01 23:32:05,256 splashto.py] setting values: [{'url':
> u'http://code.google.com/apis/ajaxsearch/documentation',
> 'thumbnail_url': u'http://images.websnapr.com/?
> size=S&key=69wD353Q05qN&url=http%3A%2F%2Fcode.google.com%2Fapis
> %2Fajaxsearch%2Fdocumentation', 'title': u'Developer'sGuide -
> Google AJAX Search API - Google Code'}, {'url': u'http://
> facebook.com', 'thumbnail_url': u'http://images.websnapr.com/?
> size=S&key=69wD353Q05qN&url=http%3A%2F%2Ffacebook.com', 'title':
> u'http://facebook.com'}]
>
> ERROR 2009-02-01 23:32:05,336 __init__.py] maximum recursion depth
> exceeded
> Traceback (most recent call last):
> File "/usr/local/lib/google_appengine/google/appengine/ext/webapp/
> __init__.py", line 498, in __call__
> handler.get(*groups)
> File "/home/matthew/code/splashTo/splashto.py", line 39, in get
> splashes = get_splashes(key)
> File "/home/matthew/code/splashTo/splashto.py", line 66, in
> get_splashes
> memcache.add(cache_key, splashes, 60)
> File "/usr/local/lib/google_appengine/google/appengine/api/memcache/
> __init__.py", line 557, in add
> return self._set_with_policy(MemcacheSetRequest.ADD, key, value,
> time=time)
> File "/usr/local/lib/google_appengine/google/appengine/api/memcache/
> __init__.py", line 602, in _set_with_policy
> stored_value, flags = _validate_encode_value(value,
> self._do_pickle)
> File "/usr/local/lib/google_appengine/google/appengine/api/memcache/
> __init__.py", line 170, in _validate_encode_value
> stored_value = do_pickle(value)
> File "/usr/local/lib/google_appengine/google/appengine/api/memcache/
> __init__.py", line 274, in DoPickle
> self._pickler_instance.dump(value)
> File "/usr/lib/python2.5/pickle.py", line 224, in dump
> self.save(obj)
> File "/usr/lib/python2.5/pickle.py", line 286, in save
> f(self, obj) # Call unbound method with explicit self
> ...
> (this goes on for hundreds of lines)
> ...
> File "/usr/lib/python2.5/pickle.py", line 419, in save_reduce
> save(state)
> File "/usr/lib/python2.5/pickle.py", line 286, in save
> f(self, obj) # Call unbound method with explicit self
> File "/usr/lib/python2.5/pickle.py", line 649, in save_dict
> self._batch_setitems(obj.iteritems())
> File "/usr/lib/python2.5/pickle.py", line 681, in _batch_setitems
> save(v)
> File "/usr/lib/python2.5/pickle.py", line 286, in save
> f(self, obj) # Call unbound method with explicit self
> File "/usr/lib/python2.5/pickle.py", line 725, in save_inst
> save(stuff)
> File "/usr/lib/python2.5/pickle.py", line 286, in save
> f(self, obj) # Call unbound method with explicit self
> File "/usr/lib/python2.5/pickle.py", line 649, in save_dict
> self._batch_setitems(obj.iteritems())
> File "/usr/lib/python2.5/pickle.py", line 681, in _batch_setitems
> save(v)
> File "/usr/lib/python2.5/pickle.py", line 286, in save
> f(self, obj) # Call unbound method with explicit self
> File "/usr/lib/python2.5/pickle.py", line 496, in save_unicode
> self.write(BINUNICODE + pack(" RuntimeError: maximum recursion depth exceeded
>
> @@@
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appengine@googlegroups.com
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en
-~--~~~~--~~--~--~---
[google-appengine] Stuck indexes
Could someone please clear my stuck indexes? I have an index that has been building for many hours now and only a few entities. I did try vacuuming without success. AppID is "webnodes" Thanks P.S. Are there any plans to automate the resetting of indexes? I've seen many similar posts of users who have the same problem. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: How to define unique constraint for properties?
You could add a property to your BlogName model which would link to the canonical BlogName. However, you'd also have to take this into account when editing blog names, since there is always a trade off between features and complexity. Chris On May 20, 8:59 pm, keakon wrote: > Anyone knows how to define unique constraint in datastore? > > See I have a Blog model, and it has these properties: > name, content, title, keywords, description and so on. > > I need provide a URL to get this blog like this: > http:///blog/.html > > So user cannot have two blogs that share the same name, otherwise one > of them cannot be gotten via URL. > > I also need provide a way to change blogs' names. > > It seems I need to define a BlogName model to store Blog's name as > keyname, and start a transaction to store the 2 entities: > > Creating: > 1. Get BlogName entity by keyname, if not exsit, create one, else exit > as failed. > 2. Create a Blog entity which parent and name is this BlogName entity. > > Editing: > 1. Get BlogName entity by keyname, if not exsit, create one, else exit > as failed. > 2. Get the old Blog entity and old BlogName entity by keyname. > 3. Create a Blog entity which parent and name is this BlogName entity, > and rest the same to the old one. > 4. Delete the old 2 ones. > > Well, it seems can work, but how terrible it is! > > And if I want a blog can have multiple names, so the two URLs can get > the same blog: > http:///blog/hello.html > http:///blog/你好.html > > I don't think I can get them by keyname since they are in a list > property now, so it can't be done in a transaction. > And if I keep them in two Blog entities, I need change both of them > when I edit one of its properties. > So it seems I need define BlogContent model, BlogTitle model, > BlogKeywords model, BlogDescription model for referece from Blog > model. > > Any better idea for me? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: ReferenceProperty with key_name ?
I wrote some custom properties awhile back which may help: http://gist.github.com/109193 They are untested, so proceed with caution. On Jun 7, 12:29 am, Tom Wu wrote: > Hi All, > > Since the db.key is auto generate by system. We can't assign the value of > db.key. > > If ReferenceProperty can reference with key_name, it will make easier for us > to backup and restore the Reference model. > > Just a idea for me, Any suggestion will be appreciated. > > Best Regards > Tom Wu --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
[google-appengine] Re: *help* app engine down?
Looks like appengine.google.com is down at the moment. On Nov 18, 8:50 am, "Alexander W." wrote: > Hi. > > My app is down. All requests result in error code 500. Even, the app > engine dashboard shows a 500 error. > I tried deploying a new version which also didn't work. > > @AppEngineTeam: please take a look at this. > > appID: sexyornotapp -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
