[google-appengine] Re: 1.5.2 SDK Prerelease
For developers facing this issue locally, it may be useful to know that I only was able to access the old local datastore by adding the flag --default_partition= (without any quotes). -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/RAuu4omiwwUJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
Suggestion: When you prerelease, could you stick that in the eclipse repository too, for ease of download? Brandon Donnelson http://gwt-examples.googlecode.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/5cCk5uJvnykJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
I too am extremely interested to see an in-depth explanation of not requiring an exploding index. My project was shelved a year ago because of the exploding index issue. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/mL7xRViiWusJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Hi Alfred, Thanks for clarification. I also have 2 questions: 1. For this query: SELECT * FROM Model WHERE list = :1 AND list =:2 AND list=:3 AND string :=4 ORDER BY date DESC will be enough this index? - kind: Model properties: - name: list - name: string - name: date direction: desc 2. After I create this index (or indexes) I will still receive this exceptions? com.google.appengine.api.datastore.DatastoreNeedIndexException The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query.. An index is missing but we are unable to tell you which one due to a bug in the App Engine SDK. If your query only contains equality filters you most likely need a composite index on all the properties referenced in those filters. --Alex -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/cHCKy8QEXw0J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
On Fri, Jul 15, 2011 at 12:25 AM, Alexandru Farcaş alex.far...@expert-group.biz wrote: Hi Alfred, Thanks for clarification. I also have 2 questions: 1. For this query: SELECT * FROM Model WHERE list = :1 AND list =:2 AND list=:3 AND string :=4 ORDER BY date DESC will be enough this index? - kind: Model properties: - name: list - name: string - name: date direction: desc Yes. This is the index the SDK will now suggest. 2. After I create this index (or indexes) I will still receive this exceptions? com.google.appengine.api.datastore.DatastoreNeedIndexException The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query.. An index is missing but we are unable to tell you which one due to a bug in the App Engine SDK. If your query only contains equality filters you most likely need a composite index on all the properties referenced in those filters. It is possible. This means that there are lots of results that match each filter and no results that match all filters (in the first 10k results). If you see this, adding the following exploding index should help a great deal: - kind: Model properties: - name: list - name: list - name: string - name: date We plan on removing this exception in the future, but this won't improve the efficiency of the query (the only thing that will do that is adding indexes like this). --Alex -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/cHCKy8QEXw0J. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-python] Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Yes, you are correct and those indexes will work. It's a trade of, composite indexes 'pre-intersect' (at write time) properties while zigzag merge join 'post-intersects' properties (at read time). I left the ancestor in because it is probably very 'selective' which has the potential to greatly reduce the amount of data that needs intersected at read time (though this is very data dependent). On Thu, Jul 14, 2011 at 11:35 AM, PK p...@gae123.com wrote: Alfred thanks for the clarification. However, isn't ancestor a list too that could contribute to an explosion (albeit minor assuming shallow hierarchies). If this is the case, would these indexes help/work? - kind: Model ancestor: yes properties: - name: int - name: date direction: desc - kind: Model properties: - name: list1 - name: int - name: date direction: desc - kind: Model properties: - name: list2 - name: int - name: date direction: desc Thanks -- You received this message because you are subscribed to the Google Groups google-appengine-python group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-python/-/aQh0Xx49xlsJ. To post to this group, send email to google-appengine-pyt...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-python+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-python?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
OMG... finally... zigzag merge join... Any info on query performance SELECT * FROM Model WHERE list = :1 AND list = :2 AND list = :3 ORDER BY date DESC with index - kind: Model properties: - name: list - name: date or is it highly dependent on data distribution? Matija -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/w_WoWIWmEncJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-python] Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
:-) performance is data dependent. Here is convoluted explanation of performance: Sx = set of entities where list = :x smallest_set = min(S1.size(), S2.size(), ...) It works best when the intersection(S1, S2, S3,...) is large compared to the smallest_set. The pathological case is intersection(S1, S2, S3, ...) = 0 and smallest_set = |all data| / 2 On Fri, Jul 15, 2011 at 12:26 PM, Matija matija.jerko...@gmail.com wrote: OMG... finally... zigzag merge join... Any info on query performance SELECT * FROM Model WHERE list = :1 AND list = :2 AND list = :3 ORDER BY date DESC with index - kind: Model properties: - name: list - name: date or is it highly dependent on data distribution? Matija -- You received this message because you are subscribed to the Google Groups google-appengine-python group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-python/-/w_WoWIWmEncJ. To post to this group, send email to google-appengine-pyt...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-python+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-python?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Hi Alfred, Thanks for clarification. I also have 2 questions: 1. For this query: SELECT * FROM Model WHERE list = :1 AND list =:2 AND list=:3 AND string :=4 ORDER BY date DESC will be enough this index? - kind: Model properties: - name: list - name: string - name: date direction: desc 2. After I create this index (or indexes) I will still receive this exceptions? com.google.appengine.api.datastore.DatastoreNeedIndexException The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query.. An index is missing but we are unable to tell you which one due to a bug in the App Engine SDK. If your query only contains equality filters you most likely need a composite index on all the properties referenced in those filters. --Alex -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/cHCKy8QEXw0J. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
On Fri, Jul 15, 2011 at 12:25 AM, Alexandru Farcaş alex.far...@expert-group.biz wrote: Hi Alfred, Thanks for clarification. I also have 2 questions: 1. For this query: SELECT * FROM Model WHERE list = :1 AND list =:2 AND list=:3 AND string :=4 ORDER BY date DESC will be enough this index? - kind: Model properties: - name: list - name: string - name: date direction: desc Yes. This is the index the SDK will now suggest. 2. After I create this index (or indexes) I will still receive this exceptions? com.google.appengine.api.datastore.DatastoreNeedIndexException The built-in indices are not efficient enough for this query and your data. Please add a composite index for this query.. An index is missing but we are unable to tell you which one due to a bug in the App Engine SDK. If your query only contains equality filters you most likely need a composite index on all the properties referenced in those filters. It is possible. This means that there are lots of results that match each filter and no results that match all filters (in the first 10k results). If you see this, adding the following exploding index should help a great deal: - kind: Model properties: - name: list - name: list - name: string - name: date We plan on removing this exception in the future, but this won't improve the efficiency of the query (the only thing that will do that is adding indexes like this). --Alex -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/cHCKy8QEXw0J. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-python] Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Yes, you are correct and those indexes will work. It's a trade of, composite indexes 'pre-intersect' (at write time) properties while zigzag merge join 'post-intersects' properties (at read time). I left the ancestor in because it is probably very 'selective' which has the potential to greatly reduce the amount of data that needs intersected at read time (though this is very data dependent). On Thu, Jul 14, 2011 at 11:35 AM, PK p...@gae123.com wrote: Alfred thanks for the clarification. However, isn't ancestor a list too that could contribute to an explosion (albeit minor assuming shallow hierarchies). If this is the case, would these indexes help/work? - kind: Model ancestor: yes properties: - name: int - name: date direction: desc - kind: Model properties: - name: list1 - name: int - name: date direction: desc - kind: Model properties: - name: list2 - name: int - name: date direction: desc Thanks -- You received this message because you are subscribed to the Google Groups google-appengine-python group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-python/-/aQh0Xx49xlsJ. To post to this group, send email to google-appengine-pyt...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-python+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-python?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
I filed this enhancement request earlier today, if you also see this need as you experiment with indexes please star it: http://code.google.com/p/googleappengine/issues/detail?id=5346 Building indexes for large data sets is expensive so deleting and recreating them from scratch needs to be avoided. With NextGen queries features the need arises to experiment with different index strategies for functionality and performance reasons. What I propose is a mechanism---probably through the admin interface---to disable an index for query purposes. The index must still be updated but is not used for queries until it is enabled again. When it is enabled the index is fully functional without any rebuilding overhead. This way a user can put all the possible indexes in her index.yaml file, enable/disable some of them in production and perform tests/measurements. When satisfied she can vacuum the unnecessary ones. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/wjGQgHTQa3gJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
OMG... finally... zigzag merge join... Any info on query performance SELECT * FROM Model WHERE list = :1 AND list = :2 AND list = :3 ORDER BY date DESC with index - kind: Model properties: - name: list - name: date or is it highly dependent on data distribution? Matija -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/w_WoWIWmEncJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-python] Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
:-) performance is data dependent. Here is convoluted explanation of performance: Sx = set of entities where list = :x smallest_set = min(S1.size(), S2.size(), ...) It works best when the intersection(S1, S2, S3,...) is large compared to the smallest_set. The pathological case is intersection(S1, S2, S3, ...) = 0 and smallest_set = |all data| / 2 On Fri, Jul 15, 2011 at 12:26 PM, Matija matija.jerko...@gmail.com wrote: OMG... finally... zigzag merge join... Any info on query performance SELECT * FROM Model WHERE list = :1 AND list = :2 AND list = :3 ORDER BY date DESC with index - kind: Model properties: - name: list - name: date or is it highly dependent on data distribution? Matija -- You received this message because you are subscribed to the Google Groups google-appengine-python group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-python/-/w_WoWIWmEncJ. To post to this group, send email to google-appengine-pyt...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-python+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-python?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
I should point out the SDK is currently very insistent about it's suggestions and believes that everything else is wrong (will throw an NeedIndexError), even though it may not be. We are working making the SDK smarter in this regards, but until then you will have to test in production (you can test in production now). Here is a short blurb about how you can remove exploding indexes (official docs coming soon): Consider: SELECT * FROM Model WHERE list1 = :1 AND list2 =:2 AND ancestor is :3 AND int = :4 ORDER BY date DESC Which 'needs' this index (from the perspective of the SDK): - kind: Model ancestor: yes properties: - name: list1 - name: list2 - name: int - name: date direction: desc Here a easy way to figure out what non-exploding indexes can be used instead: 1. Group all properties with equality/ancestor filters 1. in this case [ancestor, list1, list2, int] 2. Split this grouping such that no multi-valued properties are in the same group 1. one example is: [ancestor, list1], [list2, int], but the most efficient split actually repeats the single value properties: [ancestor, list1, int], [ancestor, list2, int] 3. Create the following indexes For each group: add index([grouped values] + [query orders/inequality]) in this case: index(ancestor, list1, int, -date) index(ancestor, list2, int, -date) or - kind: Model ancestor: yes properties: - name: list1 - name: int - name: date direction: desc - kind: Model ancestor: yes properties: - name: list2 - name: int - name: date direction: desc - Alfred On Tue, Jul 12, 2011 at 4:34 PM, Alfred Fuller arfuller+appeng...@google.com wrote: Hi, It means that there are alternatives to using exploding indexes (i.e. they are no longer required to execute a given query). You can still have them (there are cases where they are useful, namely to optimize query speed over write cost) and the SDK will still suggest them in many cases (as it is hard to detect them by looking at the query). However, the SDK will no long suggest indexes with the same property repeated multiple times (as these are obviously an exploding index). One example where this is very important is SearchableModelhttp://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/search/__init__.py, which previously was crippled by exploding indexes if you tried to sort results (and now works as expected). We are working on an article that goes through how to decided when to remove or keep them, but for now the Google IO talk from 2010, Next Gen Querieshttp://www.google.com/events/io/2010/sessions/next-gen-queries-appengine.html (The Zigzag Merge Join += Sort part), is a good resource if you want a really deep dive on how this works. - Alfred On Tue, Jul 12, 2011 at 2:24 AM, Pascal Voitot Dev pascal.voitot@gmail.com wrote: exactly the same question ;) On Tue, Jul 12, 2011 at 11:21 AM, Max thebb...@gmail.com wrote: Great job! May I know more about *t**he datastore now never requires an exploding index*? Does that mean we don't need to build exploding index or simply can't build exploding index? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KyL9f70-VtkJ. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
congratulations on this Alfred. I will try out a filter on multiple words in the same list. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/GF_idAeRSMsJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
Still no fulltext search? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/AQOMSH-8ZbkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Alfred thanks for the clarification. However, isn't ancestor a list too that could contribute to an explosion (albeit minor assuming shallow hierarchies). If this is the case, would these indexes help/work? - kind: Model ancestor: yes properties: - name: int - name: date direction: desc - kind: Model properties: - name: list1 - name: int - name: date direction: desc - kind: Model properties: - name: list2 - name: int - name: date direction: desc Thanks -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/aQh0Xx49xlsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
congratulations on this Alfred. I will try out a filter on multiple words in the same list. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/GF_idAeRSMsJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Thanks Alfred, For Datastore Plus project, is there (or will there be) a Java version? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/A0gRhzlMY9MJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
A lot of great long expected features. Thanks!! I just started to experiment and the dev~ prefix is breaking scripts and processes I have been using. What motivated this change? Thanks PK -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/BCjFNxXdqKcJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
It seams the explanation to The datastore now never requires an exploding index is incomplete. Can you give an example that illustrates the difference? (before/now ) On Jul 13, 11:06 am, PK p...@gae123.com wrote: A lot of great long expected features. Thanks!! I just started to experiment and the dev~ prefix is breaking scripts and processes I have been using. What motivated this change? Thanks PK -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
Hi PK, The dev~ prefix is there to more closely replicate what happens for many apps running on the App Engine servers: they have a s~ in the APPLICATION_ID environment variable. (Specifically, the ones running on the High Replication Datastore). This will help developers identify where they're relying on it. You should use the new (and possibly not yet included in our docs--if not, it will be soon) app_identity API. from google.appengine.api import app_identity appid = app_identity.get_application_id() Is this what is causing you issues? As a workaround (or if you want to simulate the behavior of the Master/ Slave datastore), you can use --default_partition= on the command line, but you should move your application to use the API method. Also note that if you have existing information in your dev appserver datastore, it will appear to have flushed when the full appid changes (from appid to dev~appid); you can use the flag to keep it as before, or use the bulkloader to dump/restore across app ids. --Matthew On Jul 13, 1:06 am, PK p...@gae123.com wrote: A lot of great long expected features. Thanks!! I just started to experiment and the dev~ prefix is breaking scripts and processes I have been using. What motivated this change? Thanks PK -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
On Jul 12, 11:16 pm, Max thebb...@gmail.com wrote: Thanks Alfred, For Datastore Plus project, is there (or will there be) a Java version? No, NDB is Python specific in many ways. Java already has a decent async datastore API based on Futures: http://code.google.com/appengine/docs/java/datastore/async.html --Guido van Rossum (not Alfred, but Datastore Plus' author :-) -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
Matthew, thanks for the reply. In our tests we use remote API extensively. The same datastore is updated from either the server or remote API code. The remote API scripts need to be given the appid and we have an environment variable defining it. Of course this is now broken. I am retreating to a mechanism that says if you are operating agains the dev datastore then the APPID is dev~ + real_app_id else it is real_app_id, I am hopeful this will fix the issues. Between working around this issue and the further regression of 3643 (see my comment there) I have not yet been able to pass our automated tests with 1.5.2_prerelease Anyway, I am optimistic, and look forward to starting simplifying some of my indexes next. I have been waiting for more than a year for that :-) Thanks, PK -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/IM5nJU9Uo8sJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
Does transaction refer to database transactions? Or does multiple concurrent transactions mean asynchronous operations in general? This is not clear to me. Does it support asynchronous urlfetch now? And, in a slightly related question, are there plans to raise the limit of simultaneous asynchronous urlfetch calls on production? It's currently limited to 10. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Datastore transactions On Wed, Jul 13, 2011 at 2:51 PM, pdknsk pdk...@googlemail.com wrote: Does transaction refer to database transactions? Or does multiple concurrent transactions mean asynchronous operations in general? This is not clear to me. Does it support asynchronous urlfetch now? And, in a slightly related question, are there plans to raise the limit of simultaneous asynchronous urlfetch calls on production? It's currently limited to 10. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
Hi PK, You shouldn't do any sort of special checks. Instead, use the API. In the case of remote_api, don't specify the appid at all. Instead, specify the endpoint (e.g. a.appspot.com/_ah/remote_api or localhost: 8080/_ah/remote_api). The tool will look up the app id. --Matthew On Jul 13, 1:51 pm, PK p...@gae123.com wrote: Matthew, thanks for the reply. In our tests we use remote API extensively. The same datastore is updated from either the server or remote API code. The remote API scripts need to be given the appid and we have an environment variable defining it. Of course this is now broken. I am retreating to a mechanism that says if you are operating agains the dev datastore then the APPID is dev~ + real_app_id else it is real_app_id, I am hopeful this will fix the issues. Between working around this issue and the further regression of 3643 (see my comment there) I have not yet been able to pass our automated tests with 1.5.2_prerelease Anyway, I am optimistic, and look forward to starting simplifying some of my indexes next. I have been waiting for more than a year for that :-) Thanks, PK -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
exactly the same question ;) On Tue, Jul 12, 2011 at 11:21 AM, Max thebb...@gmail.com wrote: Great job! May I know more about *t**he datastore now never requires an exploding index*? Does that mean we don't need to build exploding index or simply can't build exploding index? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KyL9f70-VtkJ. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Hi, It means that there are alternatives to using exploding indexes (i.e. they are no longer required to execute a given query). You can still have them (there are cases where they are useful, namely to optimize query speed over write cost) and the SDK will still suggest them in many cases (as it is hard to detect them by looking at the query). However, the SDK will no long suggest indexes with the same property repeated multiple times (as these are obviously an exploding index). One example where this is very important is SearchableModelhttp://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/search/__init__.py, which previously was crippled by exploding indexes if you tried to sort results (and now works as expected). We are working on an article that goes through how to decided when to remove or keep them, but for now the Google IO talk from 2010, Next Gen Querieshttp://www.google.com/events/io/2010/sessions/next-gen-queries-appengine.html (The Zigzag Merge Join += Sort part), is a good resource if you want a really deep dive on how this works. - Alfred On Tue, Jul 12, 2011 at 2:24 AM, Pascal Voitot Dev pascal.voitot@gmail.com wrote: exactly the same question ;) On Tue, Jul 12, 2011 at 11:21 AM, Max thebb...@gmail.com wrote: Great job! May I know more about *t**he datastore now never requires an exploding index*? Does that mean we don't need to build exploding index or simply can't build exploding index? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KyL9f70-VtkJ. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
Great job! May I know more about *t**he datastore now never requires an exploding index *? Does that mean we don't need to build exploding index or simply can't build exploding index? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KyL9f70-VtkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: 1.5.2 SDK Prerelease
exactly the same question ;) On Tue, Jul 12, 2011 at 11:21 AM, Max thebb...@gmail.com wrote: Great job! May I know more about *t**he datastore now never requires an exploding index*? Does that mean we don't need to build exploding index or simply can't build exploding index? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KyL9f70-VtkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
Hi, Thanks for the prerelease! Is nice to start playing with the new features for the new pricing model. Could you clarify about the exploding indexes change? is not clear to me either. Cheers, Sebastian www.devsar.com -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/Z3WZ9KzcBSIJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: 1.5.2 SDK Prerelease
The SDK now supports multiple concurrent transactions. What does this mean exactly? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [appengine-java] Re: [google-appengine] Re: 1.5.2 SDK Prerelease
Hi, It means that there are alternatives to using exploding indexes (i.e. they are no longer required to execute a given query). You can still have them (there are cases where they are useful, namely to optimize query speed over write cost) and the SDK will still suggest them in many cases (as it is hard to detect them by looking at the query). However, the SDK will no long suggest indexes with the same property repeated multiple times (as these are obviously an exploding index). One example where this is very important is SearchableModelhttp://code.google.com/p/googleappengine/source/browse/trunk/python/google/appengine/ext/search/__init__.py, which previously was crippled by exploding indexes if you tried to sort results (and now works as expected). We are working on an article that goes through how to decided when to remove or keep them, but for now the Google IO talk from 2010, Next Gen Querieshttp://www.google.com/events/io/2010/sessions/next-gen-queries-appengine.html (The Zigzag Merge Join += Sort part), is a good resource if you want a really deep dive on how this works. - Alfred On Tue, Jul 12, 2011 at 2:24 AM, Pascal Voitot Dev pascal.voitot@gmail.com wrote: exactly the same question ;) On Tue, Jul 12, 2011 at 11:21 AM, Max thebb...@gmail.com wrote: Great job! May I know more about *t**he datastore now never requires an exploding index*? Does that mean we don't need to build exploding index or simply can't build exploding index? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/KyL9f70-VtkJ. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine for Java group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Re: [google-appengine] Re: 1.5.2 SDK Prerelease
The Datastore itself has always supported multiple concurrent transactions. However the dev_appserver in the python SDK previously used a global lock and would deadlock if a single thread tried to start more than a single transaction. Now you can have multiple concurrent transactions running at the same time. This is good for the Go runtime (which uses the python dev_appserver), the Datastore Plushttp://code.google.com/p/appengine-ndb-experiment/ library, anyone else who wants to use concurrent transactions on the dev_appserver. On Tue, Jul 12, 2011 at 3:19 PM, pdknsk pdk...@googlemail.com wrote: The SDK now supports multiple concurrent transactions. What does this mean exactly? -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups Google App Engine group. To post to this group, send email to google-appengine@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
