Re: [google-appengine] Roadmap for Datastore features described in "Next Gen Queries" (I/O 2010 talk)?

We released a version of most of these in the search API: https://developers.google.com/appengine/docs/python/search/ We also released open source code that can be used to do the last two https://code.google.com/p/s2-geometry-library-java/ That being said, we are still planning to make these first

Re: [google-appengine] Re: Not Equal Operator in Filter with datastore.Query in GAE with Python

recommend you file a feature request to support this use case so we can track a good solution. On Fri, Oct 11, 2013 at 3:26 AM, Mitul Golakiya wrote: > Thanks Alftred, It worked... > > Thank you so much for you help... > > > > > On Friday, October 11, 2013 1:22:22 PM UTC+5

Re: [google-appengine] Re: Not Equal Operator in Filter with datastore.Query in GAE with Python

class DynamicEntity(db.Expando): > @classmethod > def _get_kind(cls): > return 'MyCustomEntity' > > newEntity = DynamicEntity() > newEntity.fname = "Test" > newEntity.put() > > > &

Re: [google-appengine] Re: Not Equal Operator in Filter with datastore.Query in GAE with Python

FYI, the following function should be sufficient to register a dynamic kind in the kind_map: def addExpandoForKind(kind): class Dummy(db.Expando): @classmethod def _get_kind(cls): return kind though it is not a "good" solution :-) (there are all sorts of memory, thread and request

Re: [google-appengine] Not Equal Operator in Filter with datastore.Query in GAE with Python

; Thanks Alfred... > > To use both ndb or Expando class, first I have to create a class which > extends ndb or Expando... > But I am getting entity name at run time, So I do not have any fixed > entity. > > Our users can store their own entities at run time. > Suppose I am creati

Re: [google-appengine] Not Equal Operator in Filter with datastore.Query in GAE with Python

FYI both ndb and db support dynamic properties through the Expando class On Wed, Oct 9, 2013 at 5:36 PM, Vinny P wrote: > On Wed, Oct 9, 2013 at 8:32 AM, Mitul Golakiya > wrote: > >> We are using datastore.py for querying data from datastore, because we >> have to define our entities at runti

Re: [google-appengine] Re: Upcoming Change to Datastore Auto IDs

2^53 is the largest exact *integer* value representable by a 64-bit *float* * * scattered ids are guaranteed to be representable by a 64-bit float which means they are less than or equal to 2^53. On Fri, Jun 14, 2013 at 11:12 AM, Nico Verwer wrote: > I should have said: 2^53 is the largest exac

Re: [google-appengine] Re: Upcoming Change to Datastore Auto IDs

A Java long is a 64-bit integer and the correct primitive to store an ID in. On Thu, Jun 13, 2013 at 2:15 AM, jon wrote: > > Also, in response to Ray and Timh regarding integer data types: although >> the new auto ids will be too large to store in 32-bit integers, javascript >> code which enco

Re: [google-appengine] Scatters with NDB

The dev app server populates the scatter index at 50% as it is assumed to have much less data. On Tue, May 22, 2012 at 8:18 PM, Kaan Soral wrote: > Unfortunately I didn't get the chance to, probably should set a dummy > project and code etc for this kind of stuff > > I decided to add my own sca

Re: [google-appengine] Re: Datastore OFFSET is WORTHLESS

wrote: > Also in Java API? > > On Monday, October 22, 2012 8:33:00 PM UTC+2, Alfred Fuller wrote: > >> Backwards paging is currently supported, you can see this ndb example of >> how to do it: https://developers.google.**com/appengine/docs/python/ndb/* &

Re: [google-appengine] Re: Datastore OFFSET is WORTHLESS

Backwards paging is currently supported, you can see this ndb example of how to do it: https://developers.google.com/appengine/docs/python/ndb/queries#cursors On Wed, Oct 10, 2012 at 12:29 AM, Matija wrote: > Hey, > is there any news about backwards cursor? Two years ago they were > 'presented'

Re: [google-appengine] App deployment stuck and then failed with "version not ready"

oyment > is really painful. > > > On Thursday, September 13, 2012 4:43:05 PM UTC-7, Alfred Fuller wrote: > >> It is possible that command line tool gave up too soon and it just taking >> longer than normal to deploy the new app version. In this case the version >>

Re: [google-appengine] App deployment stuck and then failed with "version not ready"

issue, it becomes intermittent for me now. For about >> 30% time, I am able to finish the deployment. I will try to create a new >> environment and try to deploy a sample app there. >> >> On Thursday, September 13, 2012 11:35:21 AM UTC-7, Alfred Fuller wrote: >>>

Re: [google-appengine] App deployment stuck and then failed with "version not ready"

On Thu, Sep 13, 2012 at 9:19 AM, gks wrote: > With the introduction of OR query, GAE SDK 1.7+ marks Query.setFilter as > deprecated so that upgrading to 1.7 will lead to a lot of warnings in my > project. Is backward compatibility guaranteed in 1.7.1 with those warning? The warnings are safe to

Re: [google-appengine] Re: Datastore Indexes stuck. vacuum_indexes didn't fix it

Are you using the low level API? Can you give me the .toString() of the query? On Tue, Aug 21, 2012 at 9:20 AM, Oleg Bashtenko wrote: > Got another problem. > There is a "Ticket" kind in my datastore. I want to run a search query on > a several fields, e.g.: { active, technicianKey, vip, dueDate

Re: [google-appengine] HR datastore batch put

n Monday, March 14, 2011 6:52:55 PM UTC+1, Alfred Fuller wrote: >> >> Batch put is not atomic. When using a batch put across entity groups it >> is very important to make sure the operation is idempotent (for example by >> not using auto id allocation) as retrying a fa

Re: [google-appengine] About the new OR-queries (1.7.0)

True, thx takashi. On Thu, Jun 28, 2012 at 7:36 AM, Takashi Matsuo wrote: > Currently, it's the latter for now, and billed per sub-query. We're > building something better than the current implementation(as always). > > -- Takashi > > > > On Thu, Jun 28, 2012 at 1:17 AM, Joakim wrote: > >> Dur

Re: [google-appengine] Re: Task Queue API: Pricing Risk?

It doesn't use the datastore or charge datastore ops/storage. On Jun 22, 2012 6:09 PM, "Waleed Abdulla" wrote: > I wonder if the task queue is using the datastore behind the scenes and > the cost of those calls are included in the datastore quota? Have you > checked to see if using the task queu

Re: [google-appengine] Projection Queries does not work with order() and Limit parameter

App Engine handles index building for you. When you add a new index, it will re-index all entities. However, it is possible to save entities with an 'unindexed' property, then change the model definition to save the same property but 'indexed.' In this case, the old entities will remain unindexed,

Re: [google-appengine] Projection Queries does not work with order() and Limit parameter

thing to check would be if the missing records have values for >> the properties specified in the projection. >> >> Rgds >> >> T >> >> On Tuesday, June 19, 2012 10:35:32 PM UTC+8, Christopher Ramírez wrote: >>> >>> Yes it happens on both se

Re: [google-appengine] Projection Queries does not work with order() and Limit parameter

Does this happen in both the dev_appserver and production? On Fri, Jun 15, 2012 at 10:28 AM, Christopher Ramírez < blindedbythed...@gmail.com> wrote: > I was trying to use projection queries, but I had extrange issues using > them. > > I'm trying to use them like this: > offset_group = Client

Re: [google-appengine] Re: Datastore Backup / Restore by namespace. Is scripting on servers such as Amazon the only solution ?

We are working hard on improving the built-in backup experience, but in the meantime: The datastore admin is open source so you can make the backup MR do what every you want: http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/datastore_admin/backup_handler.py

Re: [google-appengine] Re: Write Ops incurred during Model.put()

t from the cost of an update). On Wed, Jun 6, 2012 at 1:16 PM, mac wrote: > Hi Alfred, > > Sorry for bringing this old topic back, I am also confused by the number > showed in the dev_appserver. > > I think you meant > 1 op for the entity > 2x2 = 4 ops for each changed in

Re: [google-appengine] GAE Python DB change Broke My App!

/api/datastore.py#1086). Are you passing in an empty string to the decode function? Anything that produces an expando should have failed to encode and decode before (with an exception). Do you have a byte string ("print '%r' % encoded_proto") you can send me for testing? Tha

Re: [google-appengine] Migration to HRD stuck in Sync phase (read only mode)

We have found and fixed the problem (it was a utf-8 encoding issue with the blob metadata). If you try again it should work. Thanks for your patients. On Sat, Apr 14, 2012 at 6:52 AM, Lucian Baciu wrote: > I'm trying to migrate my app (id: timetonote) to HRD. However, it seems is > stuck at 66.6

Re: [google-appengine] App Engine 1.6.5 Released

ndb does not yet officially support projection queries, so there might be some rough edges if you use them. On Apr 25, 2012 1:23 AM, "Gopal Patel" wrote: > does ndb supports projection query ? > > On Wed, Apr 25, 2012 at 11:43 AM, Alfred Fuller wrote: > >> I agr

Re: [google-appengine] Re: App Engine 1.6.5 Released

Projection queries return the properties you projected and key for each entity that matches the query. On Wed, Apr 25, 2012 at 10:39 AM, pdknsk wrote: > When using projection queries for just a single property, does this > also return keys? The answer is probably yes, as returned items can > pro

Re: [google-appengine] App Engine 1.6.5 Released

I agree :-), the doc update is in progress. On Tue, Apr 24, 2012 at 10:41 PM, Sun Jing wrote: > Thank you, it works. I think the document should be updated: > > https://developers.google.com/appengine/docs/python/datastore/queryclass#Query_get > > -- > You received this message because you are s

Re: [google-appengine] App Engine 1.6.5 Released

You can pass the projection in as a keyword argument to any of the query functions: Article.all().run/fetch/get(projection=('title',)) On Tue, Apr 24, 2012 at 9:58 PM, Sun Jing wrote: > Thank you, Alfred. > Can I change projection if I use Model.all() to create a Query obje

Re: [google-appengine] App Engine 1.6.5 Released

Oh, also db.Query takes a class not a string, should be: db.Query(Article, projection=('title',)).get() On Tue, Apr 24, 2012 at 9:09 PM, Alfred Fuller wrote: > ('title') is not a tuple, you need to use ('title',) or ['title'] instead > >

Re: [google-appengine] App Engine 1.6.5 Released

('title') is not a tuple, you need to use ('title',) or ['title'] instead >>> print ('title') title >>> print ('title',) ('title',) >>> On Tue, Apr 24, 2012 at 8:58 PM, Sun Jing wrote: > I tried to use projection queries: > > class Article(db.Model): > title = db.StringProperty(required=True) >

Re: [google-appengine] Re: 1.6.5 Prerelease SDKs now available!

For now you have to hunt around in the doc string, look for "projection" near query objects :-). On Sat, Apr 21, 2012 at 11:13 AM, Nathaniel Stensland wrote: > How to find out more about > > - You can now perform datastore queries that return a subset of your entity > properties with the same p

Re: [google-appengine] Re: The API package 'remote_socket' or call 'CreateSocket()' was not found.

It looks like the problem might be: "Do not add Oracle®'s JavaMail JARs to your app; if you do, the app will throw exceptions." I see a bunch of "com.sun.mail.smtp..." in the stack trace. On Fri, Apr 20, 2012 at 8:01 AM, Alfred Fuller wrote: > Hmm, I thi

Re: [google-appengine] Re: The API package 'remote_socket' or call 'CreateSocket()' was not found.

Hmm, I think I may be completely wrong :-) https://developers.google.com/appengine/docs/java/mail/overview On Fri, Apr 20, 2012 at 7:55 AM, Alfred Fuller wrote: > She means the app engine Mail API: > http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/

Re: [google-appengine] Re: The API package 'remote_socket' or call 'CreateSocket()' was not found.

She means the app engine Mail API: http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/mail/ javax.mail is not supported. On Fri, Apr 20, 2012 at 5:06 AM, Emanuele Ziglioli < theb...@emanueleziglioli.it> wrote: > Hi Christina, > > I'm using the ma

Re: [google-appengine] Re: LOTS of errors on db.get() suddenly

Ya, this is a pickling issue, we have created a fix for it and will deploy it soon. In the mean time, recreating any queries that were pickled using the 1.6.4 runtime will fix this problem. On Wed, Apr 18, 2012 at 8:25 PM, Amy Unruh wrote: > Try clearing memcache from the admin console. (You ar

Re: [google-appengine] Re: LOTS of errors on db.get() suddenly

his is supported under > 1.6.5, but we're in an in-between stage at the moment... > > BadRequestError: keys only queries are not supported by multi-query. > > j > > On Apr 18, 3:56 pm, Alfred Fuller wrote: > > Should work for everything, if not, let me know :-) > >

Re: [google-appengine] Re: LOTS of errors on db.get() suddenly

Should work for everything, if not, let me know :-) On Wed, Apr 18, 2012 at 2:55 PM, Jason Collins wrote: > Will this work for MultiQuery too? > j > > On Apr 18, 3:50 pm, Alfred Fuller wrote: > > Is query a db.GqlQuery or a db.Query? > > > > In either case

Re: [google-appengine] Re: LOTS of errors on db.get() suddenly

Is query a db.GqlQuery or a db.Query? In either case you can use: query.run(keys_only=True) # works for fetch and get as well On Wed, Apr 18, 2012 at 2:48 PM, Jason Collins wrote: > This is almost certainly due to a change in a protected attribute on > the Query class in 1.6.5 (which is rollin

Re: [google-appengine] Migration to HRD - 4 days - Not done

It looks like you have a very large number of namespaces. This is a known issue with the GAE MR framework, we are looking into using an alternate approach to migrate your app. On Wed, Apr 11, 2012 at 6:52 AM, Aswath Satrasala < aswath.satras...@gmail.com> wrote: > Hello Chris, > The HRD migration

Re: [google-appengine] Master/Slave Datastore, thanks for all your hard work

Migrating all data is supported already (unless I am missing something). Does it say otherwise somewhere? On Mon, Apr 9, 2012 at 12:15 PM, andrew wrote: > Hi Chris, > > Migrating all data from all namespaces (I.e. a transparent migration of > all data that is then accessed by same app via same AP

Re: [google-appengine] Re: Master/Slave Datastore, thanks for all your hard work

We've already lasted 4 years, which makes us an outlier in your list :-). On Thu, Apr 5, 2012 at 4:16 PM, Andrei wrote: > will gae last 3 years? > google wave, google buzz, ... gae? > > On Apr 4, 11:20 pm, Andrei wrote: > > 3 years from now? > > > > On Apr 4, 4:42 pm, Chris Ramsdale wrote: > >

Re: [google-appengine] Re: How can i increase indexes of my application

It seems like there are two separate issues here: We have a bug where the index quota isn't being updated correctly (we are looking into this). When you have a lot of kinds you may actually need a lot of indexes. For example 200 indexes / 50 kinds = 4 indexes per kind. This is reasonable. I am

Re: [google-appengine] Re: Switched to HRD - So far not impressed

Most of the time 'first reply' works even when using strong consistency because we cache the state of entity groups in all replicas. Though to read from a replica, it has to be up to date. So setting eventual consistency will help you if an entity group is written to a lot (likely not up to date) o

Re: [google-appengine] Huge number of datastore reads?

What is your app id? Did you perform a lot of queries using remote api? On Tue, Mar 6, 2012 at 4:40 PM, Dave Peck wrote: > My production application went over its daily budget today. Let's just > say that my daily budget is roughly 3x what I've ever actually needed > on my busiest day. Today wa

Re: [google-appengine] Using data migration tool for production bug reproduction

We currently have no plans to expand the feature set of the migration tool to fit alternative use cases. That being said, the "datastore admin" page does provide a copy to app feature that should come close to meeting the needs of your use case. You can also use datastore triggers ( http://code.go

Re: [google-appengine] Re: HRD migration tool now available!

Did you alias your old app id to your new app id? Cause if that is the case, then remote_api is likely talking to the new app (and I don't think it is possible to make it talk to the old app). On Mon, Dec 12, 2011 at 9:56 AM, MikeR wrote: > Hi Simon, > > Thanks for the reply; I'm not expecting a

Re: [google-appengine] Datastore Small Operations and Nov 7th

The new pricing model has always included count() and our accounting code didn't change at launch. What is your app id so I can take a deeper look? On Sat, Nov 26, 2011 at 3:14 AM, Millisecond wrote: > It appears that on the same day billing was enabled (Nov 7th), what > counted as a "Datastore

Re: [google-appengine] HRD migration tool now available!

onment variables there)? > > On Nov 13, 2011, at 2:54 PM, Alfred Fuller wrote: > > full_app_id<https://cs.corp.google.com/#google3/apphosting/api/app_identity/app_identity.py&ct=xref_usages&gs=py:google3.apphosting.api.app_identity.app_identity.get_application_id&full_app_id:go

Re: [google-appengine] HRD migration tool now available!

full_app_id= os

Re: [google-appengine] Re: 1.6.0 is now launched

Can you be more specific about what extra entities are actually being written. The SDKs now split batch get/delete/puts, are you sure you aren't just seeing that? On Tue, Nov 8, 2011 at 9:02 AM, Alexis wrote: > I've got a small "issue" with this release: > the development server is writing many

Re: [google-appengine] Re: Datastore operations are seriously expensive

Here is an article that can help you reduce the # of composite indexes you use in some cases: http://code.google.com/appengine/articles/indexselection.html On Tue, Nov 8, 2011 at 4:16 PM, Jeff Schnitzer wrote: > FWIW, your app would have been a problem under the old pricing model > as well. The

Re: [google-appengine] Re: HRD uses more datastore writes?

Wait, are you comparing the old pricing model or the new pricing model (since you attributed the cost to writes I just assumed you were talking about the new pricing model)? In the old pricing model HRD writes cost more (~3x). On Fri, Oct 21, 2011 at 8:18 AM, Jason Collins wrote: > This is making

Re: [google-appengine] Understanding Exploding Indexes

On Tue, Oct 11, 2011 at 8:50 AM, Edward Hartwell Goose wrote: > Hi, > > I'm trying to work out why some indexes I've got have gone into an error > state. > > My interpretation of the help text ( > http://code.google.com/appengine/docs/python/datastore/queries.html#Big_Entities_and_Exploding_Indexe

Re: [google-appengine] HRD uses more datastore writes?

There is no difference between M/S and HRD with regards to write ops (i.e. given the same input, M/S and HRD report the exact same amount of writes). Something else must be different (traffic, existing entity values, etc). On Tue, Oct 18, 2011 at 12:46 PM, Jason Collins wrote: > I had a M-S app t

[google-appengine] Re: [appengine-python] Re: Prerelease SDK 1.5.5 available for download!

XG transactions will have no effect on global queries, as the fundamental problems still remains (namely that it is impossible to know what entity groups will/should appear in a global query). Additionally you should not use this read_policy, it has little effect in M/S and no effect in HRD. On W

Re: [google-appengine] Re: Zig zag merge feature request

It will only suggest better indexes in some cases (specifically the cases where the same property was repeated in multiple equality filters, e. g. WHERE tag=:1 AND tag=:2 ORDER BY date) On Wed, Sep 28, 2011 at 1:53 PM, jay wrote: > Thanks for the reply Jason. > > So you are saying that the autom

[google-appengine] Re: Zig zag merge feature request

onitor the performance over time). - Alfred On Sep 26, 8:08 am, Jason Collins wrote: > We've been working on a "billing optimization project" for our large > app, steprep. Right now, we are tackling index optimizations - > datastore writes form the largest part of our b

Re: [google-appengine] Re: Write Ops incurred during Model.put()

i > is presumably much cheaper if only a few properties are changed? > > On Tue, Sep 20, 2011 at 12:27 PM, Simon Knott > wrote: > > Hi, > > > > According to Alfred, who's a Googler who appears to know lots about the > > datastore, only the updated values cause

Re: [google-appengine] Re: Worst-case scenario for eventual consistency in the HRD?

very very unlikely very quickly, but not impossible). There really is no hard upper bounds because distributed systems will have pieces that fail (and are designed to still function when they do). - Alfred On Tue, Sep 20, 2011 at 10:10 AM, Ikai Lan (Google) wrote: > Well, indexes are just Bigta

Re: [google-appengine] Re: Worst-case scenario for eventual consistency in the HRD?

An interesting notion. Although you could also just use ColorThings(key_name=color) as the parent entity for all the Things. This way the list of things would be queriable directly (using an ancestor query) and there would not be a limit on the number and size of Things. They also exist next to eac

Re: [google-appengine] Help Testing with --high_replication

Wed, Sep 7, 2011 at 2:53 PM, Alfred Fuller wrote: > We were able to reproduce it. Something strange is going on. I'll let you > know what we figure out. > > > On Wed, Sep 7, 2011 at 2:03 PM, Alfred Fuller < > arfuller+appeng...@google.com> wrote: > >> Ok,

Re: [google-appengine] Help Testing with --high_replication

We were able to reproduce it. Something strange is going on. I'll let you know what we figure out. On Wed, Sep 7, 2011 at 2:03 PM, Alfred Fuller wrote: > Ok, this is a very different problem then. > > Here is the function that reads the data: > http://code.google.com/p/googl

Re: [google-appengine] Help Testing with --high_replication

Ok, this is a very different problem then. Here is the function that reads the data: http://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/api/datastore_file_stub.py#444 Here is the function that writes the data: http://code.google.com/p/googleappengine/source/browse

Re: [google-appengine] Help Testing with --high_replication

writes? Thanks, Alfred On Wed, Sep 7, 2011 at 11:44 AM, Joshua Smith wrote: > I think you are mistaken that calling Flush will help. > > (I don't have the first clue how to do that. My python skills are not up > to understanding the stuff in ext.testbed) > > I added this lin

Re: [google-appengine] Help Testing with --high_replication

t that point. Something very strange > is happening there. > > Bye, > Waldemar > > On Tue, Sep 6, 2011 at 11:20 PM, Alfred Fuller > wrote: > > Hmm. I have seen it work this way on my system. This is the function you > > need have > > invoked > http://code.

Re: [google-appengine] Re: Large number of datastore puts

value for each index) deleting the entity will cost the same a creating it (9 write ops) It is important to note that these costs have not changed in the new pricing model, they were just obfuscated in the old model in the form of cpu hours. Hope this clears it up. - Alfred On Tue, Sep 6, 2011 at

Re: [google-appengine] Help Testing with --high_replication

t; record is gone. > > Note that ^C sends sigINT which is the same thing the STOP sign on the > launcher sends. > > On Sep 6, 2011, at 1:45 PM, Alfred Fuller wrote: > > You would have to have code that only exists in your dev environment (or > just never gets called in produc

Re: [google-appengine] Help Testing with --high_replication

the server. It is probably much much easier to just run the dev_appserver manually and use ctrl-c to kill it. On Tue, Sep 6, 2011 at 10:27 AM, Joshua Smith wrote: > How do I call Flush() on the stub manually? Do I do this in my main.py > someplace, somehow? > > On Sep 6, 2011, at 1:2

Re: [google-appengine] Help Testing with --high_replication

t 10:10 AM, Joshua Smith wrote: > I've updated to 1.5.3, but I still have this problem. > > How do I "shut down gracefully"? I'm shutting down by pressing the big red > STOP button on the launcher. > > Is there another way? > > -Joshua > > On Sep 6, 2011

Re: [google-appengine] Help Testing with --high_replication

If the dev_appserver is shut down gracefully, then you should not see this problem (as of 1.5.3). On Tue, Sep 6, 2011 at 8:40 AM, Joshua Smith wrote: > When I start with --high_replication to test on the dev appserver, I alway > seem to be starting with a blank database. > > I found this issue: >

Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude

BTW, I really like the blog post Emlyn. On Sun, Sep 4, 2011 at 11:54 PM, Alfred Fuller < arfuller+appeng...@google.com> wrote: > > > On Sat, Sep 3, 2011 at 8:23 AM, johnP wrote: > >> >> Which leads to a potential constructive suggestion. Maybe Goog can >&g

Re: [google-appengine] Re: The Amazing Story Of Appengine And The Two Orders Of Magnitude

On Sat, Sep 3, 2011 at 8:23 AM, johnP wrote: > > Which leads to a potential constructive suggestion. Maybe Goog can > post a troubleshooting guide that lists different line-items in the > new-style billing, and potential gotchas? An example in your blog is > that sudden parallelism is costly an

Re: [google-appengine] Re: Abysmal Query Performance in Backend

On Sun, Sep 4, 2011 at 1:31 PM, Volker Schönefeld < volker.schoenef...@gmail.com> wrote: > Hey Alfred, > > thanks for the reply. I've got a couple more questions regarding the > compaction, I hope you can answer a few: > > - When do those compaction events happen? >

Re: [google-appengine] Re: Abysmal Query Performance in Backend

The datastore does not delete things right away. It marks them as delete and waits for a "compaction" to actually remove the data. If a lot of data is deleted at the start of a query, the datastore will have to skip all the deleted rows until it finds the first real entity. This is what is causing

Re: Re: Aw: Re: [google-appengine] Re: Small Datastore Operations = $50.00 per day??

as a sharded counter). We have not built these solutions directly into the datastore because an application's tolerance for inaccuracy can vary drastically (no one size fits all solution). Thanks, Alfred On Thu, Sep 1, 2011 at 1:05 PM, Daniel Florey wrote: > Thanks a lot for your answer. >

Re: [google-appengine] Re: Serious problem: Rollback of data on HRD

and thanks for zigzag merge join BTW, it rocks) > Tom > > On Aug 19, 9:34 pm, Alfred Fuller > wrote: > > Are your transactions idempotent? It is possible that the transaction is > > being run (and succeeding) twice in this case. What other request is > > colliding

Re: [google-appengine] Re: Serious problem: Rollback of data on HRD

Are your transactions idempotent? It is possible that the transaction is being run (and succeeding) twice in this case. What other request is colliding with first? You are not using any non-ancestor queries or setting read_policy=EVENTUAL on any reads correct? On Fri, Aug 19, 2011 at 12:42 PM, Tom

Re: [appengine-python] Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease

:-) performance is data dependent. Here is convoluted explanation of performance: Sx = set of entities where list = :x smallest_set = min(S1.size(), S2.size(), ...) It works best when the intersection(S1, S2, S3,...) is large compared to the smallest_set. The pathological case is intersection(S1

Re: [appengine-python] Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease

which has the potential to greatly reduce the amount of data that needs intersected at read time (though this is very data dependent). On Thu, Jul 14, 2011 at 11:35 AM, PK wrote: > Alfred thanks for the clarification. > > However, isn't ancestor a list too that could contribute

Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease

On Fri, Jul 15, 2011 at 12:25 AM, Alexandru Farcaş < alex.far...@expert-group.biz> wrote: > Hi Alfred, > > Thanks for clarification. I also have 2 questions: > > 1. For this query: > SELECT * FROM Model WHERE list = :1 AND list =:2 AND list=:3 AND string :=4 > ORDER BY

Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease

name: list1 - name: int - name: date direction: desc - kind: Model ancestor: yes properties: - name: list2 - name: int - name: date direction: desc - Alfred On Tue, Jul 12, 2011 at 4:34 PM, Alfred Fuller < arfuller+appeng...@google.com> wrote: > Hi, > > It me

Re: [google-appengine] Re: 1.5.2 SDK Prerelease

Datastore transactions On Wed, Jul 13, 2011 at 2:51 PM, pdknsk wrote: > Does transaction refer to database transactions? Or does "multiple > concurrent transactions" mean asynchronous operations in general? This > is not clear to me. Does it support asynchronous urlfetch now? > > And, in a sligh

Re: [google-appengine] Re: 1.5.2 SDK Prerelease

The Datastore itself has always supported multiple concurrent transactions. However the dev_appserver in the python SDK previously used a global lock and would deadlock if a single thread tried to start more than a single transaction. Now you can have multiple concurrent transactions running at the

Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease

m 2010, Next Gen Queries<http://www.google.com/events/io/2010/sessions/next-gen-queries-appengine.html> (The "Zigzag Merge Join += Sort" part), is a good resource if you want a really deep dive on how this works. - Alfred On Tue, Jul 12, 2011 at 2:24 AM, Pascal Voitot Dev < pa

Re: [google-appengine] Re: Alfred Fuller - "... So we've done that ..." : ZigZag merge join += Sort

Still working on it :-). On Fri, Jul 8, 2011 at 3:58 PM, acuth wrote: > ...or is it to be superceded by the Full Text Search service they detailed > at Google I/O 2011? http://www.youtube.com/watch?v=7B7FyU9wW8Y > > -- > You received this message because you are subscribed to the Google Groups >

[google-appengine] Re: BadRequestError: query not found (occurring in the middle of a for loop)

responding in http://code.google.com/p/googleappengine/issues/detail?id=4432 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/8BVe0TVks5gJ. To post

Re: [google-appengine] Updated App Engine Pricing FAQ!

appear in). Hopefully the new pricing model is clear enough that you don't feel inclined to dig that deep. On Sun, Jun 26, 2011 at 12:37 PM, Stephen wrote: > On Sun, Jun 26, 2011 at 5:02 PM, Alfred Fuller > wrote: > > > > - Additional small op charge per key fetched

Re: [google-appengine] Updated App Engine Pricing FAQ!

On Sat, Jun 25, 2011 at 5:36 PM, Jeff Schnitzer wrote: > On Sat, Jun 25, 2011 at 3:26 PM, Alfred Fuller < > arfuller+appeng...@google.com> wrote: > >> This makes sense, and encourages more use of memcache. to hold entities. >>> One question that I've be

Re: [google-appengine] Updated App Engine Pricing FAQ!

entities vs 500 entities part of this question, which is hard to predict. > > Also, is there any way we can get the transaction timestamp out on > datastore writes? This would *dramatically* improve the robustness of code > that tries to keep memcache in sync with the datastore dur

Re: [google-appengine] Simple query times out repeatedly for hours!

ing the raw query every so often to catch them (or keep a minimum eta in memcache, or a maximum eta to eliminate clock skew issues, etc) 3. Shard the ETA index so you can have several processers running not reading/removing the same index values or locations - Alfred On Wed, Jun 22,

Re: [google-appengine] What does the new --high_replication flag actually do?

the dev server you will never see an old value once a new value is seen (not so in production) - Alfred On Wed, Jun 22, 2011 at 10:01 AM, Robert Kluin wrote: > Hi Gwyn, > It basically adds a small delay before a transaction applies, but > with facilities to grab consistent snapshots

Re: [google-appengine] Re: Is the native API really so much faster than JDO and slim3?

44 AM, Alfred Fuller < arfuller+appeng...@google.com> wrote: > Hmm. The one on code.google.com seems to be out of date. The version > shipping with the SDK has: > > private final transient > QueryResultIteratorImpl<https://cs.corp.google.com/#google3/java/com/g

Re: [google-appengine] Re: Is the native API really so much faster than JDO and slim3?

amp;ct=xref_jump_to_def&l=340>(); out <https://cs.corp.google.com/#google3/java/com/google/appengine/api/datastore/LazyList.java&ct=xref_jump_to_def&l=354>.defaultWriteObject <http://www.google.com/url?sa=D&q=http%3A%2F%2Fjava.sun.com%2Fjavase%2F6%2Fdocs%2Fapi%2Fjava%2Fio%2FObjec

Re: [google-appengine] Re: Is the native API really so much faster than JDO and slim3?

to do something similar in Objectify. > > Thanks for clearing this up! > > Jeff > > On Thu, Jun 9, 2011 at 10:29 AM, Alfred Fuller > wrote: > > If you are going to just iterate through a list with out doing any work, > > fetching everything up front is always go

Re: [google-appengine] Re: Is the native API really so much faster than JDO and slim3?

ad of doing the conversion on demand) will negate any benefit from async prefetching. Also a realistic benchmark should tak into account that some amount of work will be done on the entities fetched. On Thu, Jun 9, 2011 at 9:51 AM, Alfred Fuller wrote: > It does uses a lazy list to do asynch

Re: [google-appengine] Re: Is the native API really so much faster than JDO and slim3?

It does uses a lazy list to do asynchronous prefetching: http://code.google.com/p/googleappengine/source/browse/trunk/java/src/main/com/google/appengine/api/datastore/LazyList.java On Tue, Jun 7, 2011 at 3:19 AM, Anders wrote: > I doubt that the difference can be that large. The performance test

Re: [google-appengine] Re: BadRequestError: cursor position cannot specify start inclusivity with out a start key

onsolation the new cursor format is significantly smaller and much more robust to future changes (and support some really cool features that will come out soon ;-)). Thanks, Alfred P.S. If you have seen it recently in production please let me know. On Mon, May 9, 2011 at 5:03 AM, Maarten wrote

Re: [google-appengine] Re: BadRequestError: cursor position cannot specify start inclusivity with out a start key

It will be the same for your app, but different for other apps. Should be fixed in the order of minutes. On Mon, May 2, 2011 at 2:21 PM, dloomer wrote: > Are you getting the same exact (invalid) query.cursor() value I get? i.e. > E-ABABQ= > > -- > You received this message because you are subsc

[google-appengine] Re: BadRequestError: cursor position cannot specify start inclusivity with out a start key

Looking into it -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more o

  1   2   >