[appengine-java] Re: custom vs. generated keys / storage?
The encoded "Entity Key" is a representation that includes your key
name, entity kind, ancestors and app id - encoded into a form that is
safe to use in urls etc. Keeping your keys small and kinds small
(i.e. using short class names) will reduce storage space of keys in
the DS. In Twig (i.e. not JDO) you can keep the names of your classes
as you like and then override typetoKind(Type) to return abbreviated
names. I found that this saves a significant amount of space in my
app that has many relations between entities.
http://code.google.com/p/twig-persist/
On Jan 4, 5:01 pm, phraktle wrote:
> Hi,
>
> I'm generating my own entity keys, with new Entity("Foo", "myId"). I
> noticed that in the DataStore viewer, these entities still seem to
> have a default key:
>
> Decoded entity key: Foo: name=myId
> Entity key:
> ahFzY2FyYWJyZWNvbW1lbmRlcnJACxIVUHJvZHVjdFJlY29tbWVuZGF0aW9uIiU1OTg1MEMzMzM
> wOUZGRjV8MjAxMDAxMDQwMTAzNTN8cHJvZF8wDA
>
> So the "Entity key" here still seems to be generated by GAE. Why is
> this needed? I do have a primary key already. So does it get stored as
> simply another indexed column? Does this impact storage space too? I
> don't need the GAE entity key at all, and would prefer not having to
> pay for its storage either :)
>
> Thanks,
> Viktor
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: custom vs. generated keys / storage?
Hi,
I'm using the low-level DS API, thus short names for kinds, etc, and
my keys are already quite compact. The encoded version is in fact
significantly longer than my own key. So it would optimal, if DS did
not try to second-guess and bloat my PKs. How do you know that DS
doesn't actually store both versions? Do you know how to decode the DS
key?
Regards,
Viktor
On Jan 5, 5:54 am, jd wrote:
> The encoded "Entity Key" is a representation that includes your key
> name, entity kind, ancestors and app id - encoded into a form that is
> safe to use in urls etc. Keeping your keys small and kinds small
> (i.e. using short class names) will reduce storage space of keys in
> the DS. In Twig (i.e. not JDO) you can keep the names of your classes
> as you like and then override typetoKind(Type) to return abbreviated
> names. I found that this saves a significant amount of space in my
> app that has many relations between entities.
>
> http://code.google.com/p/twig-persist/
>
> On Jan 4, 5:01 pm, phraktle wrote:
>
>
>
> > Hi,
>
> > I'm generating my own entity keys, with new Entity("Foo", "myId"). I
> > noticed that in the DataStore viewer, these entities still seem to
> > have a default key:
>
> > Decoded entity key: Foo: name=myId
> > Entity key:
> > ahFzY2FyYWJyZWNvbW1lbmRlcnJACxIVUHJvZHVjdFJlY29tbWVuZGF0aW9uIiU1OTg1MEMzMzM
> > wOUZGRjV8MjAxMDAxMDQwMTAzNTN8cHJvZF8wDA
>
> > So the "Entity key" here still seems to be generated by GAE. Why is
> > this needed? I do have a primary key already. So does it get stored as
> > simply another indexed column? Does this impact storage space too? I
> > don't need the GAE entity key at all, and would prefer not having to
> > pay for its storage either :)
>
> > Thanks,
> > Viktor
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: custom vs. generated keys / storage?
could someone from Google weigh in on this?
thanks,
Viktor
On Jan 6, 12:43 pm, phraktle wrote:
> Hi,
>
> I'm using the low-level DS API, thus short names for kinds, etc, and
> my keys are already quite compact. The encoded version is in fact
> significantly longer than my ownkey. So it would optimal, if DS did
> not try to second-guess and bloat my PKs. How do you know that DS
> doesn't actually store both versions? Do you know how to decode the DSkey?
>
> Regards,
> Viktor
>
> On Jan 5, 5:54 am, jd wrote:
>
>
>
> > The encoded "EntityKey" is a representation that includes yourkey
> > name, entity kind, ancestors and app id - encoded into a form that is
> > safe to use in urls etc. Keeping your keys small and kinds small
> > (i.e. using short class names) will reduce storage space of keys in
> > the DS. In Twig (i.e. not JDO) you can keep the names of your classes
> > as you like and then override typetoKind(Type) to return abbreviated
> > names. I found that this saves a significant amount of space in my
> > app that has many relations between entities.
>
> >http://code.google.com/p/twig-persist/
>
> > On Jan 4, 5:01 pm, phraktle wrote:
>
> > > Hi,
>
> > > I'm generating my own entity keys, with new Entity("Foo", "myId"). I
> > > noticed that in the DataStore viewer, these entities still seem to
> > > have a defaultkey:
>
> > > Decoded entitykey: Foo: name=myId
> > > Entitykey:
> > > ahFzY2FyYWJyZWNvbW1lbmRlcnJACxIVUHJvZHVjdFJlY29tbWVuZGF0aW9uIiU1OTg1MEMzMzM
> > > wOUZGRjV8MjAxMDAxMDQwMTAzNTN8cHJvZF8wDA
>
> > > So the "Entitykey" here still seems to be generated by GAE. Why is
> > > this needed? I do have a primarykeyalready. So does it get stored as
> > > simply another indexed column? Does this impact storage space too? I
> > > don't need the GAE entitykeyat all, and would prefer not having to
> > > pay for its storage either :)
>
> > > Thanks,
> > > Viktor
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
[appengine-java] Re: custom vs. generated keys / storage?
Hi Jeff,
Yeah, I get that, I'm just failing to see why this "alternative
representation" is even there and exposed in any way? The keys must be
indexed with the representation that I have constructed, as I can
perform lexicographical searches on those specific keys. So what's the
other representation all about? It seems wasteful to me. So I would
still like to understand better, what the mechanics are here...
Viktor
On Jan 9, 12:11 pm, Jeff Schnitzer wrote:
> You have a mistaken understanding of what a Key is. Both values
> displayed on the Datastore Viewer
> ("aghtb2JjYS1zdHIOCxIHQ2hlY2tpbhjiQww" and "YourEntityKind: id=12345")
> are merely alternative visual representations. The actual value
> stored in BigTable is a binary representation of this data. It may or
> may not be more compact than either of these two visual
> representations, but you can be assured that there is only one stored
> version of it (not counting foreign key references or replication, of
> course).
>
> I don't work for Google, but I watched the Google I/O video on
> BigTable. You should too.
>
> Jeff
>
>
>
> On Fri, Jan 8, 2010 at 11:07 PM, phraktle wrote:
>
> > could someone from Google weigh in on this?
>
> > thanks,
> > Viktor
>
> > On Jan 6, 12:43 pm, phraktle wrote:
> >> Hi,
>
> >> I'm using the low-level DS API, thus short names for kinds, etc, and
> >> my keys are already quite compact. The encoded version is in fact
> >> significantly longer than my ownkey. So it would optimal, if DS did
> >> not try to second-guess and bloat my PKs. How do you know that DS
> >> doesn't actually store both versions? Do you know how to decode the DSkey?
>
> >> Regards,
> >> Viktor
>
> >> On Jan 5, 5:54 am, jd wrote:
>
> >> > The encoded "EntityKey" is a representation that includes yourkey
> >> > name, entity kind, ancestors and app id - encoded into a form that is
> >> > safe to use in urls etc. Keeping your keys small and kinds small
> >> > (i.e. using short class names) will reduce storage space of keys in
> >> > the DS. In Twig (i.e. not JDO) you can keep the names of your classes
> >> > as you like and then override typetoKind(Type) to return abbreviated
> >> > names. I found that this saves a significant amount of space in my
> >> > app that has many relations between entities.
>
> >> >http://code.google.com/p/twig-persist/
>
> >> > On Jan 4, 5:01 pm, phraktle wrote:
>
> >> > > Hi,
>
> >> > > I'm generating my own entity keys, with new Entity("Foo", "myId"). I
> >> > > noticed that in the DataStore viewer, these entities still seem to
> >> > > have a defaultkey:
>
> >> > > Decoded entitykey: Foo: name=myId
> >> > > Entitykey:
> >> > > ahFzY2FyYWJyZWNvbW1lbmRlcnJACxIVUHJvZHVjdFJlY29tbWVuZGF0aW9uIiU1OTg1MEMzMzM
> >> > > wOUZGRjV8MjAxMDAxMDQwMTAzNTN8cHJvZF8wDA
>
> >> > > So the "Entitykey" here still seems to be generated by GAE. Why is
> >> > > this needed? I do have a primarykeyalready. So does it get stored as
> >> > > simply another indexed column? Does this impact storage space too? I
> >> > > don't need the GAE entitykeyat all, and would prefer not having to
> >> > > pay for its storage either :)
>
> >> > > Thanks,
> >> > > Viktor
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine for Java" group.
> > To post to this group, send email to google-appengine-j...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > google-appengine-java+unsubscr...@googlegroups.com.
> > For more options, visit this group
> > athttp://groups.google.com/group/google-appengine-java?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: custom vs. generated keys / storage?
You have a mistaken understanding of what a Key is. Both values
displayed on the Datastore Viewer
("aghtb2JjYS1zdHIOCxIHQ2hlY2tpbhjiQww" and "YourEntityKind: id=12345")
are merely alternative visual representations. The actual value
stored in BigTable is a binary representation of this data. It may or
may not be more compact than either of these two visual
representations, but you can be assured that there is only one stored
version of it (not counting foreign key references or replication, of
course).
I don't work for Google, but I watched the Google I/O video on
BigTable. You should too.
Jeff
On Fri, Jan 8, 2010 at 11:07 PM, phraktle wrote:
>
> could someone from Google weigh in on this?
>
> thanks,
> Viktor
>
> On Jan 6, 12:43 pm, phraktle wrote:
>> Hi,
>>
>> I'm using the low-level DS API, thus short names for kinds, etc, and
>> my keys are already quite compact. The encoded version is in fact
>> significantly longer than my ownkey. So it would optimal, if DS did
>> not try to second-guess and bloat my PKs. How do you know that DS
>> doesn't actually store both versions? Do you know how to decode the DSkey?
>>
>> Regards,
>> Viktor
>>
>> On Jan 5, 5:54 am, jd wrote:
>>
>>
>>
>> > The encoded "EntityKey" is a representation that includes yourkey
>> > name, entity kind, ancestors and app id - encoded into a form that is
>> > safe to use in urls etc. Keeping your keys small and kinds small
>> > (i.e. using short class names) will reduce storage space of keys in
>> > the DS. In Twig (i.e. not JDO) you can keep the names of your classes
>> > as you like and then override typetoKind(Type) to return abbreviated
>> > names. I found that this saves a significant amount of space in my
>> > app that has many relations between entities.
>>
>> >http://code.google.com/p/twig-persist/
>>
>> > On Jan 4, 5:01 pm, phraktle wrote:
>>
>> > > Hi,
>>
>> > > I'm generating my own entity keys, with new Entity("Foo", "myId"). I
>> > > noticed that in the DataStore viewer, these entities still seem to
>> > > have a defaultkey:
>>
>> > > Decoded entitykey: Foo: name=myId
>> > > Entitykey:
>> > > ahFzY2FyYWJyZWNvbW1lbmRlcnJACxIVUHJvZHVjdFJlY29tbWVuZGF0aW9uIiU1OTg1MEMzMzM
>> > > wOUZGRjV8MjAxMDAxMDQwMTAzNTN8cHJvZF8wDA
>>
>> > > So the "Entitykey" here still seems to be generated by GAE. Why is
>> > > this needed? I do have a primarykeyalready. So does it get stored as
>> > > simply another indexed column? Does this impact storage space too? I
>> > > don't need the GAE entitykeyat all, and would prefer not having to
>> > > pay for its storage either :)
>>
>> > > Thanks,
>> > > Viktor
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to google-appengine-j...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine-java+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
>
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Re: custom vs. generated keys / storage?
Please watch this in its entirety: http://www.youtube.com/watch?v=tx5gdoNpcZM
The java Key class, the python Key class, the concept of "MyEntity id
#123", and the string "aghtb2JjYS1zdHIOCxIHQ2hlY2tpbhjiQww" are all
various representations of some bit of binary data stored in the
database. Just because it prints to the screen in one particular way
doesn't mean it's stored that way.
Conceptually, an entity is identified by a Key, and you have the
option of converting this back and forth to a web-safe string. The
web UI likes to show you this string. Maybe the underlying bigtable
stores FK references natively in the funky String format, maybe not,
but that's an implementation detail that (fortunately) we don't have
to worry about.
Jeff
On Sat, Jan 9, 2010 at 9:38 PM, phraktle wrote:
>
> Hi Jeff,
>
> Yeah, I get that, I'm just failing to see why this "alternative
> representation" is even there and exposed in any way? The keys must be
> indexed with the representation that I have constructed, as I can
> perform lexicographical searches on those specific keys. So what's the
> other representation all about? It seems wasteful to me. So I would
> still like to understand better, what the mechanics are here...
>
> Viktor
>
> On Jan 9, 12:11 pm, Jeff Schnitzer wrote:
>> You have a mistaken understanding of what a Key is. Both values
>> displayed on the Datastore Viewer
>> ("aghtb2JjYS1zdHIOCxIHQ2hlY2tpbhjiQww" and "YourEntityKind: id=12345")
>> are merely alternative visual representations. The actual value
>> stored in BigTable is a binary representation of this data. It may or
>> may not be more compact than either of these two visual
>> representations, but you can be assured that there is only one stored
>> version of it (not counting foreign key references or replication, of
>> course).
>>
>> I don't work for Google, but I watched the Google I/O video on
>> BigTable. You should too.
>>
>> Jeff
>>
>>
>>
>> On Fri, Jan 8, 2010 at 11:07 PM, phraktle wrote:
>>
>> > could someone from Google weigh in on this?
>>
>> > thanks,
>> > Viktor
>>
>> > On Jan 6, 12:43 pm, phraktle wrote:
>> >> Hi,
>>
>> >> I'm using the low-level DS API, thus short names for kinds, etc, and
>> >> my keys are already quite compact. The encoded version is in fact
>> >> significantly longer than my ownkey. So it would optimal, if DS did
>> >> not try to second-guess and bloat my PKs. How do you know that DS
>> >> doesn't actually store both versions? Do you know how to decode the DSkey?
>>
>> >> Regards,
>> >> Viktor
>>
>> >> On Jan 5, 5:54 am, jd wrote:
>>
>> >> > The encoded "EntityKey" is a representation that includes yourkey
>> >> > name, entity kind, ancestors and app id - encoded into a form that is
>> >> > safe to use in urls etc. Keeping your keys small and kinds small
>> >> > (i.e. using short class names) will reduce storage space of keys in
>> >> > the DS. In Twig (i.e. not JDO) you can keep the names of your classes
>> >> > as you like and then override typetoKind(Type) to return abbreviated
>> >> > names. I found that this saves a significant amount of space in my
>> >> > app that has many relations between entities.
>>
>> >> >http://code.google.com/p/twig-persist/
>>
>> >> > On Jan 4, 5:01 pm, phraktle wrote:
>>
>> >> > > Hi,
>>
>> >> > > I'm generating my own entity keys, with new Entity("Foo", "myId"). I
>> >> > > noticed that in the DataStore viewer, these entities still seem to
>> >> > > have a defaultkey:
>>
>> >> > > Decoded entitykey: Foo: name=myId
>> >> > > Entitykey:
>> >> > > ahFzY2FyYWJyZWNvbW1lbmRlcnJACxIVUHJvZHVjdFJlY29tbWVuZGF0aW9uIiU1OTg1MEMzMzM
>> >> > > wOUZGRjV8MjAxMDAxMDQwMTAzNTN8cHJvZF8wDA
>>
>> >> > > So the "Entitykey" here still seems to be generated by GAE. Why is
>> >> > > this needed? I do have a primarykeyalready. So does it get stored as
>> >> > > simply another indexed column? Does this impact storage space too? I
>> >> > > don't need the GAE entitykeyat all, and would prefer not having to
>> >> > > pay for its storage either :)
>>
>> >> > > Thanks,
>> >> > > Viktor
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine for Java" group.
>> > To post to this group, send email to
>> > google-appengine-j...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > google-appengine-java+unsubscr...@googlegroups.com.
>> > For more options, visit this group
>> > athttp://groups.google.com/group/google-appengine-java?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine for Java" group.
> To post to this group, send email to google-appengine-j...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengine-java+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine-java?hl=en.
>
>
>
>
--
You received this message because you are subscribed to the Google
