[google-appengine] storage consumption in dashboard for local development server?
I'd like to get a rough estimate of the amount of storage consumed by the datastore of a local dev server fully populated with my set of data. Is there some straightforward way to do this? The local dashboard lacks the storage/quota statistics offered for deployed (live) applications. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: storage consumption in dashboard for local development server?
Issue filed: http://code.google.com/p/googleappengine/issues/detail?id=3001 On Mar 21, 11:21 pm, Robert Kluin wrote: > I agree, it would be very nice to have something in the local dev > server that would help estimate the space required on the production > system. > > If you make an issue for this I will star it, and I am sure a lot of > people would star it too. > > You might also look at this > article:http://code.google.com/appengine/articles/storage_breakdown.html > > Robert > > On Sun, Mar 21, 2010 at 4:25 AM, kostmo wrote: > > I'd like to get a rough estimate of the amount of storage consumed by > > the datastore of a local dev server fully populated with my set of > > data. Is there some straightforward way to do this? The local > > dashboard lacks the storage/quota statistics offered for deployed > > (live) applications. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to google-appeng...@googlegroups.com. > > To unsubscribe from this group, send email to > > google-appengine+unsubscr...@googlegroups.com. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] permutations vs Cartesian product in exploding indexes documentation
It is still unclear to me exactly how many index entries will be
produced by including multiple ListProperty's in an entity. In one
post [1], between the inquiring user and the Google rep, the
possibilities for the number of index entries included the power set,
Cartesian product, or the "number of unique combinations" (this was
the rep's answer, without mention of "permutations").
I'd like to verify that the correct combinatorics terminology is being
used in the documentation on exploding indexes [2].
Quote:
"the index table must include a row for every permutation of the
values of every property for the index"
This is followed by the example:
e2 = MyModel()
e2.x = ['red', 'blue']
e2.y = [1, 2]
"the index must store 12 property values: 2 each for the built-in
indexes on x and y, and 2 for each of the 4 permutations of x and y in
the custom index".
The example does not make me completely confident about what type of
combinatoric expression is truly being used, since the sum of the
number of permutations for each property happens to equal the
cardinality of the Cartesian product of the two properties. The
enumeration given in this post [3] seems to suggest that it is
actually the Cartesian product at work.
To clarify this issue, we could use Python 2.6 to illustrate an
example:
>>> from itertools import product, permutations
>>> x = ['red', 'blue', 'green']
>>> y = [1, 2, 3]
>>> list(permutations(x)) + list(permutations(y))
[('red', 'blue', 'green'), ('red', 'green', 'blue'), ('blue', 'red',
'green'), ('blue', 'green', 'red'), ('green', 'red', 'blue'),
('green', 'blue', 'red'), (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1),
(3, 1, 2), (3, 2, 1)]
>>> list(product(x, y))
[('red', 1), ('red', 2), ('red', 3), ('blue', 1), ('blue', 2),
('blue', 3), ('green', 1), ('green', 2), ('green', 3)]
Is the number of custom index entries that will be generated by an
entity with the two 3-valued properties given above proportional to
the sum of permutations (there are 12) or the Cartesian product (there
are 9)?
Thanks,
Karl
[1] "Efficient way to structure my data model"
http://groups.google.com/group/google-appengine/browse_thread/thread/326e90024ed53ced/d958fa6df98ba6df
[2]
http://code.google.com/appengine/docs/python/datastore/queriesandindexes.html#Big_Entities_and_Exploding_Indexes
[3] "Size of index in case of exploding index"
http://groups.google.com/group/google-appengine/browse_thread/thread/3db5a0338d77d81b/c5f1ea4fa3107b25
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: permutations vs Cartesian product in exploding indexes documentation
OK, I just watched Brett Slatkin's I/O talk [1] and he mentions "cross
product" a couple of times, so it seems that the use of the word
"permutation" in the docs is incorrect; the number of index entries is
indeed proportional to the Cartesian product, rather than
"permutations" which would lead to a factorial (n!) number of index
entries.
I've filed a doco issue here [2].
Karl
[1]
http://code.google.com/events/io/2009/sessions/BuildingScalableComplexApps.html
[2] http://code.google.com/p/googleappengine/issues/detail?id=3003
On Mar 22, 11:50 am, kostmo wrote:
> It is still unclear to me exactly how many index entries will be
> produced by including multiple ListProperty's in an entity. In one
> post [1], between the inquiring user and the Google rep, the
> possibilities for the number of index entries included the power set,
> Cartesian product, or the "number of unique combinations" (this was
> the rep's answer, without mention of "permutations").
>
> I'd like to verify that the correct combinatorics terminology is being
> used in the documentation on exploding indexes [2].
>
> Quote:
> "the index table must include a row for every permutation of the
> values of every property for the index"
> This is followed by the example:
> e2 = MyModel()
> e2.x = ['red', 'blue']
> e2.y = [1, 2]
> "the index must store 12 property values: 2 each for the built-in
> indexes on x and y, and 2 for each of the 4 permutations of x and y in
> the custom index".
>
> The example does not make me completely confident about what type of
> combinatoric expression is truly being used, since the sum of the
> number of permutations for each property happens to equal the
> cardinality of the Cartesian product of the two properties. The
> enumeration given in this post [3] seems to suggest that it is
> actually the Cartesian product at work.
>
> To clarify this issue, we could use Python 2.6 to illustrate an
> example:>>> from itertools import product, permutations
> >>> x = ['red', 'blue', 'green']
> >>> y = [1, 2, 3]
> >>> list(permutations(x)) + list(permutations(y))
>
> [('red', 'blue', 'green'), ('red', 'green', 'blue'), ('blue', 'red',
> 'green'), ('blue', 'green', 'red'), ('green', 'red', 'blue'),
> ('green', 'blue', 'red'), (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1),
> (3, 1, 2), (3, 2, 1)]>>> list(product(x, y))
>
> [('red', 1), ('red', 2), ('red', 3), ('blue', 1), ('blue', 2),
> ('blue', 3), ('green', 1), ('green', 2), ('green', 3)]
>
> Is the number of custom index entries that will be generated by an
> entity with the two 3-valued properties given above proportional to
> the sum of permutations (there are 12) or the Cartesian product (there
> are 9)?
>
> Thanks,
> Karl
>
> [1] "Efficient way to structure my data
> model"http://groups.google.com/group/google-appengine/browse_thread/thread/...
> [2]http://code.google.com/appengine/docs/python/datastore/queriesandinde...
> [3] "Size of index in case of exploding
> index"http://groups.google.com/group/google-appengine/browse_thread/thread/...
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: permutations vs Cartesian product in exploding indexes documentation
Thanks for the clarification, Nick.
Karl
On Mar 23, 5:58 am, "Nick Johnson (Google)"
wrote:
> Hi Karl,
>
> You're correct that it is indeed the cartesian product in this case - it
> produces one index entry for every unique tuple of values from the indexed
> columns.
>
> This gets slightly more complicated in the situation where the same column
> is being indexed multiple times. In that situation, the naive cartesian
> product is not used - instead, the additional constraints that the values
> are sorted in non-descending order and that no value occurs more than once
> in a tuple are added, so as to eliminate unnecessary duplicates. Eg, an
> index on (foo, foo), with a value of [1,2,3] for foo will result in the
> following entries:
>
> (1,2),(1,3),(2,3)
>
> Notably omitting the redundant entries:
>
> (2,1),(3,1),(3,2)
>
> And the invalid entries:
>
> (1,1),(2,2),(3,3)
>
> -Nick Johnson
>
>
>
> On Tue, Mar 23, 2010 at 2:22 AM, kostmo wrote:
> > OK, I just watched Brett Slatkin's I/O talk [1] and he mentions "cross
> > product" a couple of times, so it seems that the use of the word
> > "permutation" in the docs is incorrect; the number of index entries is
> > indeed proportional to the Cartesian product, rather than
> > "permutations" which would lead to a factorial (n!) number of index
> > entries.
>
> > I've filed a doco issue here [2].
>
> > Karl
>
> > [1]
> >http://code.google.com/events/io/2009/sessions/BuildingScalableComple...
> > [2]http://code.google.com/p/googleappengine/issues/detail?id=3003
>
> > On Mar 22, 11:50 am, kostmo wrote:
> > > It is still unclear to me exactly how many index entries will be
> > > produced by including multiple ListProperty's in an entity. In one
> > > post [1], between the inquiring user and the Google rep, the
> > > possibilities for the number of index entries included the power set,
> > > Cartesian product, or the "number of unique combinations" (this was
> > > the rep's answer, without mention of "permutations").
>
> > > I'd like to verify that the correct combinatorics terminology is being
> > > used in the documentation on exploding indexes [2].
>
> > > Quote:
> > > "the index table must include a row for every permutation of the
> > > values of every property for the index"
> > > This is followed by the example:
> > > e2 = MyModel()
> > > e2.x = ['red', 'blue']
> > > e2.y = [1, 2]
> > > "the index must store 12 property values: 2 each for the built-in
> > > indexes on x and y, and 2 for each of the 4 permutations of x and y in
> > > the custom index".
>
> > > The example does not make me completely confident about what type of
> > > combinatoric expression is truly being used, since the sum of the
> > > number of permutations for each property happens to equal the
> > > cardinality of the Cartesian product of the two properties. The
> > > enumeration given in this post [3] seems to suggest that it is
> > > actually the Cartesian product at work.
>
> > > To clarify this issue, we could use Python 2.6 to illustrate an
> > > example:>>> from itertools import product, permutations
> > > >>> x = ['red', 'blue', 'green']
> > > >>> y = [1, 2, 3]
> > > >>> list(permutations(x)) + list(permutations(y))
>
> > > [('red', 'blue', 'green'), ('red', 'green', 'blue'), ('blue', 'red',
> > > 'green'), ('blue', 'green', 'red'), ('green', 'red', 'blue'),
> > > ('green', 'blue', 'red'), (1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1),
> > > (3, 1, 2), (3, 2, 1)]>>> list(product(x, y))
>
> > > [('red', 1), ('red', 2), ('red', 3), ('blue', 1), ('blue', 2),
> > > ('blue', 3), ('green', 1), ('green', 2), ('green', 3)]
>
> > > Is the number of custom index entries that will be generated by an
> > > entity with the two 3-valued properties given above proportional to
> > > the sum of permutations (there are 12) or the Cartesian product (there
> > > are 9)?
>
> > > Thanks,
> > > Karl
>
> > > [1] "Efficient way to structure my data model"
>
[google-appengine] Re: How to list all entity kinds programmatically?
I tried to use the body of the get_kinds() function (after adding the line "from google.appengine.api import datastore_admin"), but I get the error: BadRequestError: app [myappname] cannot call GetSchema Karl On Mar 25, 2:19 pm, mstodd wrote: > Well, the admin interface canlistthem, so I tracked down how that's > happening, and it's in google/google_appengine/google/appengine/ext/ > admin/__init__.py > > check out get_kinds() in that file. > looks like it just uses datastore_admin from google.appengine.api > > let me know if you get stuck. > > On Mar 24, 7:46 pm, Trung wrote: > > > Hi, > > > I'd like to know whether we canprogrammaticallyknow thelistof > >entity > > KIND currently in the datastore. > > > Thanks in advance. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] Re: How to list all entity kinds programmatically?
Update:
I found success adapting the body of GetSchemaKinds() from
bulkloader.py:
def GetSchemaKinds(self):
"""Returns the list of kinds for this app."""
class KindStatError(Exception):
"""Unable to find kind stats for an all-kinds download."""
from google.appengine.ext.db import stats
global_stat = stats.GlobalStat.all().get()
if not global_stat:
raise KindStatError()
timestamp = global_stat.timestamp
kind_stat = stats.KindStat.all().filter(
"timestamp =", timestamp).fetch(1000)
kind_list = [stat.kind_name for stat in kind_stat
if stat.kind_name and not
stat.kind_name.startswith('__')]
kind_set = set(kind_list)
return list(kind_set)
On Mar 29, 6:12 pm, kostmo wrote:
> I tried to use the body of the get_kinds() function (after adding the
> line "from google.appengine.api import datastore_admin"), but I get
> the error:
>
> BadRequestError: app [myappname] cannot call GetSchema
>
> Karl
>
> On Mar 25, 2:19 pm, mstodd wrote:
>
> > Well, the admin interface canlistthem, so I tracked down how that's
> > happening, and it's in google/google_appengine/google/appengine/ext/
> > admin/__init__.py
>
> > check out get_kinds() in that file.
> > looks like it just uses datastore_admin from google.appengine.api
>
> > let me know if you get stuck.
>
> > On Mar 24, 7:46 pm, Trung wrote:
>
> > > Hi,
>
> > > I'd like to know whether we canprogrammaticallyknow thelistof
> > >entity
> > > KIND currently in the datastore.
>
> > > Thanks in advance.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-appeng...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] easily clear the datastore in a deployed application
Hi all, I'd like to share a script I wrote as a "plugin" to the admin panel. It lists the Kinds in a dropdown and allows you to queue a Task that deletes all entity of a single Kind, or all entities of all Kinds. http://code.google.com/p/jobfeed/wiki/Nuke I also posted about it on Stack Overflow: http://stackoverflow.com/questions/1062540/how-to-delete-all-datastore-in-google-app-engine/2549074#2549074 Cheers, Karl -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appeng...@googlegroups.com. To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
[google-appengine] data viewer couldn't retrieve list of Kinds
When I click on the Data Viewer in the "production" console, I keep getting the message "Oops! We couldn't retrieve your list of Kinds." This has happened often, for extended periods, over the past several days while I'm trying to bulk upload/delete. What specific conditions would cause this error, and how can it be avoided? It makes debugging difficult when the datastore cannot be examined. When I enter a GQL query in the box below the error message, I get no results for Kinds that I would expect to be in the datastore. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to google-appengine@googlegroups.com To unsubscribe from this group, send email to google-appengine+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~--~~~~--~~--~--~---
