Re: [appengine-java] Question: best practice to persist medium-large data?
On Mon, Jan 25, 2010 at 12:02 PM, John Patterson wrote:
> ... and then Twig makes it even easier! Because it supports direct
> relationships (without Keys) and can translate any type into a Blob for you
> your data model doesn't need to deal with low-level types at all
>
> So you example becomes:
>
> class Album {
> �...@key Long id;
> String name;
> List photos;
> }
>
> Notice how your data model is not polluted by domain classes
I can't resist a conversation about framework design philosophy :-)
This sort of binding (property of List) can be convenient in
some applications, and as a longtime Hibernate user I got used to
working like this. But I don't like this abstraction in AppEngine.
Yes, for some apps you might be able to remove the "pollution" of
framework classes like Key (or OKey), but it comes with a price.
A one-to-many relationship between Album and Photo has several
"standard" representations in AppEngine:
* The Photo could have a Key property pointing to Album
* The Photo could have a parent ancestor in its Key which points to the Album
* The Album entity could have a List property pointing to its Photos
Each choice has a dramatic impact on performance, what can be done in
a transaction, and how you do queries that simply cannot be glossed
over or abstracted away. Which does the List represent?
Furthermore, is List a proxy or did the photos get fetched
along with the Album? Can I serialize the Album or do I need to
detach it?
Even worse, there are also two more possible representations:
* The Photo could have a Key property (or ancestor) pointing to an
Album that does not exist
* The Album could have a List property, and some of the Keys
could point to Photos that do not exist
Maybe these are degenerate cases, maybe not, but you'll never be able
to completely avoid them. RDBMSes have transactions and referential
integrity constraints that guarantee these later two cases can't
happen. Not so in AppEngine. You're just one
DatastoreTimeoutException away from having to deal with this situation
in your code.
This is the big mess that makes JDO on Appengine so complicated,
possibly even more complicated than it is on an RDBMS. On the other
hand... if you simply expose the Key (or OKey), the developer does a
little more work but doesn't have to figure out all the configuration.
Without proxies, all entities are serializable and GWT-able without
any special consideration.
Actualy, JDO doesn't even offer all the necessary configuration
options, which is probably why you end up seeing Key a lot as
properties in JDO sample code.
So while it seems like you might be able to keep Key and other
framework classes out of your entities... for many applications I
don't think it's realistic, and for most, I don't think it's a good
idea.
IMHO, of course.
> Another feature only supported by Twig is embedded collections - this allows
> you to do a single query for Albums containing a particular photo for
> example. In JDO, Objectify or any of the others this requires multiple
> queries. You configure it with a single Embed annotation like this:
We like this feature and are assimilating it now :-)
Jeff
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Question: best practice to persist medium-large data?
On 26 Jan 2010, at 13:37, Jeff Schnitzer wrote: I can't resist a conversation about framework design philosophy :-) Oh go on then. Just a quickie. This sort of binding (property of List) can be convenient in some applications, and as a longtime Hibernate user I got used to working like this. But I don't like this abstraction in AppEngine. Yes, for some apps you might be able to remove the "pollution" of framework classes like Key (or OKey), but it comes with a price. Twig and Objectify operate at very different levels of abstraction. With Objectify you code at a lower level very aware of what is happening at the datastore level. It involves more work but, as you point out, if something goes wrong - it goes wrong in your own code where you are in a better position to handle it. With Twig you operate at a higher level that makes the persistence layer almost completely transparent. One major advantage of this is that you can change the way the data is stored (i.e. embedded or separate entity) without changing your code. Making such a change with Objectify means you need to rewrite your reference handling code yourself. A one-to-many relationship between Album and Photo has several "standard" representations in AppEngine: * The Photo could have a Key property pointing to Album * The Photo could have a parent ancestor in its Key which points to the Album * The Album entity could have a List property pointing to its Photos Each choice has a dramatic impact on performance, what can be done in a transaction, and how you do queries that simply cannot be glossed over or abstracted away. This is why you configure what type of relationship is used using: @Embed, @Entity(PARENT), @Entity(CHILD) or @Entity(INDEPENDENT) So you have the flexibility to choose configuration _without_ rewriting your code. Very important difference. Currently the first type of representation is not an option. I do want to add this as it makes very large collections that change often much more efficient. When it is added you could reconfigure your data schema by changing a single annotation. Such a change in Objectify would require the developer to rewrite their entire data layer Which does the List represent? Furthermore, is List a proxy or did the photos get fetched along with the Album? Can I serialize the Album or do I need to detach it? Yes instances are just normal POJOs so no problems serializing. I do this myself with GWT. Currently, lazy references are not supported... its a very important feature on the TODO list. Even worse, there are also two more possible representations: * The Photo could have a Key property (or ancestor) pointing to an Album that does not exist An inconsistent datastore is a problem with any framework - including Objectify. The trick is to use transactions where possible whenever working on an entity group to avoid getting this situation in the first place. * The Album could have a List property, and some of the Keys could point to Photos that do not exist As above. An exception would be thrown saying which property on what object could not be found. Maybe these are degenerate cases, maybe not, but you'll never be able to completely avoid them. RDBMSes have transactions and referential integrity constraints that guarantee these later two cases can't happen. Not so in AppEngine. You're just one DatastoreTimeoutException away from having to deal with this situation in your code. If your data is inconsistent you have a problem - whether the framework throws an exception (as in Twig) or if you receive a null back from a finder method (as in Objectify) there is really not much difference. You still have to clean up the mess. This is the big mess that makes JDO on Appengine so complicated, possibly even more complicated than it is on an RDBMS. On the other hand... if you simply expose the Key (or OKey), the developer does a little more work but doesn't have to figure out all the configuration. Without proxies, all entities are serializable and GWT-able without any special consideration. I think that for simple cases and small data models working with Keys and handling the references yourself is fine. When things get more complicated it is easier to reconfigure how the data is actually stored if you can simply change a configuration option in one place. Actualy, JDO doesn't even offer all the necessary configuration options, which is probably why you end up seeing Key a lot as properties in JDO sample code. So while it seems like you might be able to keep Key and other framework classes out of your entities... for many applications I don't think it's realistic, and for most, I don't think it's a good idea. IMHO, of course. Obviously we agree to disagree on this point :) I much prefer letting he framework handle repetitive boiler plate code with minimal configuration. Ano
Re: [appengine-java] Question: best practice to persist medium-large data?
Thanks John and Jeff for sharing the knowledge. I've just quickly scanned your project home, and I have a novice question that needs your help: Is twig or Objectify direct replacement to JDO / JPA on GAE? I just need to use your framework alone, or should use in combination with JDO / JPA support? What is about the approach implementing "general" JDO / JPA fit to Datastore? Is there a chance Google provide a specific implementation to their Datastore? Maybe someone from Google can advise on this? Thanks, -Chau On Tue, Jan 26, 2010 at 2:52 PM, John Patterson wrote: > > On 26 Jan 2010, at 13:37, Jeff Schnitzer wrote: > > > I can't resist a conversation about framework design philosophy :-) > > > Oh go on then. Just a quickie. > > > This sort of binding (property of List) can be convenient in > some applications, and as a longtime Hibernate user I got used to > working like this. But I don't like this abstraction in AppEngine. > Yes, for some apps you might be able to remove the "pollution" of > framework classes like Key (or OKey), but it comes with a price. > > > Twig and Objectify operate at very different levels of abstraction. With > Objectify you code at a lower level very aware of what is happening at the > datastore level. It involves more work but, as you point out, if something > goes wrong - it goes wrong in your own code where you are in a better > position to handle it. > > With Twig you operate at a higher level that makes the persistence layer > almost completely transparent. One major advantage of this is that you can > change the way the data is stored (i.e. embedded or separate entity) without > changing your code. Making such a change with Objectify means you need to > rewrite your reference handling code yourself. > > > A one-to-many relationship between Album and Photo has several > "standard" representations in AppEngine: > > * The Photo could have a Key property pointing to Album > > * The Photo could have a parent ancestor in its Key which points to the > Album > > * The Album entity could have a List property pointing to its Photos > > Each choice has a dramatic impact on performance, what can be done in > a transaction, and how you do queries that simply cannot be glossed > over or abstracted away. > > > This is why you configure what type of relationship is used using: @Embed, > @Entity(PARENT), @Entity(CHILD) or @Entity(INDEPENDENT) > > So you have the flexibility to choose configuration _without_ rewriting > your code. Very important difference. > > Currently the first type of representation is not an option. I do want to > add this as it makes very large collections that change often much more > efficient. When it is added you could reconfigure your data schema by > changing a single annotation. Such a change in Objectify would require the > developer to rewrite their entire data layer > > Which does the List represent? > Furthermore, is List a proxy or did the photos get fetched > along with the Album? Can I serialize the Album or do I need to > detach it? > > > Yes instances are just normal POJOs so no problems serializing. I do this > myself with GWT. > > Currently, lazy references are not supported... its a very important > feature on the TODO list. > > > Even worse, there are also two more possible representations: > > * The Photo could have a Key property (or ancestor) pointing to an > Album that does not exist > > > An inconsistent datastore is a problem with any framework - including > Objectify. The trick is to use transactions where possible whenever working > on an entity group to avoid getting this situation in the first place. > > * The Album could have a List property, and some of the Keys > could point to Photos that do not exist > > > As above. An exception would be thrown saying which property on what > object could not be found. > > Maybe these are degenerate cases, maybe not, but you'll never be able > to completely avoid them. RDBMSes have transactions and referential > integrity constraints that guarantee these later two cases can't > happen. Not so in AppEngine. You're just one > DatastoreTimeoutException away from having to deal with this situation > in your code. > > > If your data is inconsistent you have a problem - whether the framework > throws an exception (as in Twig) or if you receive a null back from a finder > method (as in Objectify) there is really not much difference. You still > have to clean up the mess. > > This is the big mess that makes JDO on Appengine so complicated, > possibly even more complicated than it is on an RDBMS. On the other > hand... if you simply expose the Key (or OKey), the developer does a > little more work but doesn't have to figure out all the configuration. > Without proxies, all entities are serializable and GWT-able without > any special consideration. > > > I think that for simple cases and small data models working with Keys and > handling the references yourself is fine. When things get mo
Re: [appengine-java] Question: best practice to persist medium-large data?
Hi Chau, yes both Objectify and Twig replace JDO/JPA. I think that Google would not release a non-standard datastore user-level API because they could then be accused by many of "vendor lock-in". When App Engine was launched there was a lot of concern about this. On 26 Jan 2010, at 16:59, Chau Huynh wrote: Thanks John and Jeff for sharing the knowledge. I've just quickly scanned your project home, and I have a novice question that needs your help: Is twig or Objectify direct replacement to JDO / JPA on GAE? I just need to use your framework alone, or should use in combination with JDO / JPA support? What is about the approach implementing "general" JDO / JPA fit to Datastore? Is there a chance Google provide a specific implementation to their Datastore? Maybe someone from Google can advise on this? Thanks, -Chau On Tue, Jan 26, 2010 at 2:52 PM, John Patterson wrote: On 26 Jan 2010, at 13:37, Jeff Schnitzer wrote: I can't resist a conversation about framework design philosophy :-) Oh go on then. Just a quickie. This sort of binding (property of List) can be convenient in some applications, and as a longtime Hibernate user I got used to working like this. But I don't like this abstraction in AppEngine. Yes, for some apps you might be able to remove the "pollution" of framework classes like Key (or OKey), but it comes with a price. Twig and Objectify operate at very different levels of abstraction. With Objectify you code at a lower level very aware of what is happening at the datastore level. It involves more work but, as you point out, if something goes wrong - it goes wrong in your own code where you are in a better position to handle it. With Twig you operate at a higher level that makes the persistence layer almost completely transparent. One major advantage of this is that you can change the way the data is stored (i.e. embedded or separate entity) without changing your code. Making such a change with Objectify means you need to rewrite your reference handling code yourself. A one-to-many relationship between Album and Photo has several "standard" representations in AppEngine: * The Photo could have a Key property pointing to Album * The Photo could have a parent ancestor in its Key which points to the Album * The Album entity could have a List property pointing to its Photos Each choice has a dramatic impact on performance, what can be done in a transaction, and how you do queries that simply cannot be glossed over or abstracted away. This is why you configure what type of relationship is used using: @Embed, @Entity(PARENT), @Entity(CHILD) or @Entity(INDEPENDENT) So you have the flexibility to choose configuration _without_ rewriting your code. Very important difference. Currently the first type of representation is not an option. I do want to add this as it makes very large collections that change often much more efficient. When it is added you could reconfigure your data schema by changing a single annotation. Such a change in Objectify would require the developer to rewrite their entire data layer Which does the List represent? Furthermore, is List a proxy or did the photos get fetched along with the Album? Can I serialize the Album or do I need to detach it? Yes instances are just normal POJOs so no problems serializing. I do this myself with GWT. Currently, lazy references are not supported... its a very important feature on the TODO list. Even worse, there are also two more possible representations: * The Photo could have a Key property (or ancestor) pointing to an Album that does not exist An inconsistent datastore is a problem with any framework - including Objectify. The trick is to use transactions where possible whenever working on an entity group to avoid getting this situation in the first place. * The Album could have a List property, and some of the Keys could point to Photos that do not exist As above. An exception would be thrown saying which property on what object could not be found. Maybe these are degenerate cases, maybe not, but you'll never be able to completely avoid them. RDBMSes have transactions and referential integrity constraints that guarantee these later two cases can't happen. Not so in AppEngine. You're just one DatastoreTimeoutException away from having to deal with this situation in your code. If your data is inconsistent you have a problem - whether the framework throws an exception (as in Twig) or if you receive a null back from a finder method (as in Objectify) there is really not much difference. You still have to clean up the mess. This is the big mess that makes JDO on Appengine so complicated, possibly even more complicated than it is on an RDBMS. On the other hand... if you simply expose the Key (or OKey), the developer does a little more work but doesn't have to figure out all the configuration. W
Re: [appengine-java] Question: best practice to persist medium-large data?
There aren't current plans to create yet another API for datastore operations. That's why we created the datastore API - so folks could build convenience functions as needed. I don't see anything wrong with mixing JDO/JPA with another tool for datastore access, other than adding unnecessary complexity to your application. You can already mix JDO and JPA, though you'll get confused real quick. You'll want to be careful, though, since some third-party libraries may reuse annotations. It's easy to get confused. As far as the vendor lock-in point goes: that's about the last thing we want developers to worry about. There's an open source project that aims to recreate App Engine's datastore that can be run outside our cloud. I can't quite remember the name of it, but I'll post the project when it comes to mind. On Tue, Jan 26, 2010 at 1:59 AM, Chau Huynh wrote: > Thanks John and Jeff for sharing the knowledge. > I've just quickly scanned your project home, and I have a novice question > that needs your help: > Is twig or Objectify direct replacement to JDO / JPA on GAE? I just need to > use your framework alone, or should use in combination with JDO / JPA > support? > > What is about the approach implementing "general" JDO / JPA fit to > Datastore? Is there a chance Google provide a specific implementation to > their Datastore? > Maybe someone from Google can advise on this? > > Thanks, > -Chau > > > On Tue, Jan 26, 2010 at 2:52 PM, John Patterson wrote: > >> >> On 26 Jan 2010, at 13:37, Jeff Schnitzer wrote: >> >> >> I can't resist a conversation about framework design philosophy :-) >> >> >> Oh go on then. Just a quickie. >> >> >> This sort of binding (property of List) can be convenient in >> some applications, and as a longtime Hibernate user I got used to >> working like this. But I don't like this abstraction in AppEngine. >> Yes, for some apps you might be able to remove the "pollution" of >> framework classes like Key (or OKey), but it comes with a price. >> >> >> Twig and Objectify operate at very different levels of abstraction. With >> Objectify you code at a lower level very aware of what is happening at the >> datastore level. It involves more work but, as you point out, if something >> goes wrong - it goes wrong in your own code where you are in a better >> position to handle it. >> >> With Twig you operate at a higher level that makes the persistence layer >> almost completely transparent. One major advantage of this is that you can >> change the way the data is stored (i.e. embedded or separate entity) without >> changing your code. Making such a change with Objectify means you need to >> rewrite your reference handling code yourself. >> >> >> A one-to-many relationship between Album and Photo has several >> "standard" representations in AppEngine: >> >> * The Photo could have a Key property pointing to Album >> >> * The Photo could have a parent ancestor in its Key which points to the >> Album >> >> * The Album entity could have a List property pointing to its Photos >> >> Each choice has a dramatic impact on performance, what can be done in >> a transaction, and how you do queries that simply cannot be glossed >> over or abstracted away. >> >> >> This is why you configure what type of relationship is used using: @Embed, >> @Entity(PARENT), @Entity(CHILD) or @Entity(INDEPENDENT) >> >> So you have the flexibility to choose configuration _without_ rewriting >> your code. Very important difference. >> >> Currently the first type of representation is not an option. I do want to >> add this as it makes very large collections that change often much more >> efficient. When it is added you could reconfigure your data schema by >> changing a single annotation. Such a change in Objectify would require the >> developer to rewrite their entire data layer >> >> Which does the List represent? >> Furthermore, is List a proxy or did the photos get fetched >> along with the Album? Can I serialize the Album or do I need to >> detach it? >> >> >> Yes instances are just normal POJOs so no problems serializing. I do this >> myself with GWT. >> >> Currently, lazy references are not supported... its a very important >> feature on the TODO list. >> >> >> Even worse, there are also two more possible representations: >> >> * The Photo could have a Key property (or ancestor) pointing to an >> Album that does not exist >> >> >> An inconsistent datastore is a problem with any framework - including >> Objectify. The trick is to use transactions where possible whenever working >> on an entity group to avoid getting this situation in the first place. >> >> * The Album could have a List property, and some of the Keys >> could point to Photos that do not exist >> >> >> As above. An exception would be thrown saying which property on what >> object could not be found. >> >> Maybe these are degenerate cases, maybe not, but you'll never be able >> to completely avoid them. RDBMSes have transactions and referentia
Re: [appengine-java] Question: best practice to persist medium-large data?
On Mon, Jan 25, 2010 at 11:52 PM, John Patterson wrote:
>
> This is why you configure what type of relationship is used using: @Embed,
> @Entity(PARENT), @Entity(CHILD) or @Entity(INDEPENDENT)
> So you have the flexibility to choose configuration _without_ rewriting your
> code. Very important difference.
The problem is that it *isn't* as simple as just changing an
annotation. Not in appengine, at any rate. It works in the simple
case (good for demos and sample apps), but you start to notice edge
cases:
* In some representations you can add a Photo to an Album in a
transaction, in some representations you can't.
* In some representations, multiple queries are required to fetch a
fetch an Album containing a Photo.
* Each representation has a completely different query syntax.
Twig exposes the datastore Query object, which means the developer
gets the full brunt of this exposure.
I don't want to say that it isn't possible to build a system that
abstracts entities into a sophisticated object graph - clearly, we
have JDO & JPA. What I'm saying is that the people who created JDO &
JPA are not idiots (although I do think the creators of JDO's
annotations are aesthetically challenged). The reason JDO has all
that complexity and endless configuration and query languages and
fetch groups and proxies and detaching and whatnot is because that's
what you need to abstract an arbitrary entity graph.
> Currently the first type of representation is not an option. I do want to
> add this as it makes very large collections that change often much more
> efficient. When it is added you could reconfigure your data schema by
> changing a single annotation. Such a change in Objectify would require the
> developer to rewrite their entire data layer
You can never just reconfigure your data schema with a single
annotation, both for the reasons above and because you probably have
real-world data to migrate. And real-world constraints demand a
particular schema!
Let's actually answer the original poster's question - how do you
model a photo album? I hope he's still listening :-)
The first question is how you should model it in the datastore? I'll
use Objectify's syntax here because it corresponds directly to the
datastore representation.
Actually, let's start by describing how you SHOULDN'T model a photo album.
-
BAD IDEA #1:
class Album {
@Id Long id;
String name;
List> photos;
}
class Photo {
@Id Long id;
String caption;
String blobStoreKey;// key to GAE's blobstore
}
Fetching photos in an album (again, Objectify syntax but equivalent to
the datastore operation) is:
List fetched = ofy.get(album.photos);
There are two reasons why this is a bad idea:
1) You now have a hard limit of 5,000 photos per album, established by GAE.
2) Every time you load an Album, you must load the entire set of
Photo keys. Want to generate a list of Album names? You have to load
all that key data, orders of mangitude more data than what you want.
-
BAD IDEA #2:
class Album {
@Id Long id;
String name;
}
class Photo {
@Id Long id;
@Parent OKey album;
String caption;
String blobStoreKey;
}
This stores the Photo with the Album embedded in the Photo's Key as an
ancestor, making Photo part of the Album's entity group. At first
glance, this seems kinda cool and you can now do transactions across
Albums and Photos.
Fetching photos in an album:
OQuery query = createQuery(Photo.class).ancestor(albumKey);
List fetched = ofy.prepare(query).asList();
The problem is what happens when you want to move a Photo from one
Album to another. You can't just change the parent. You must delete
the Photo entity and create a whole new Photo entity with the new
parent Album. And if the Photo has Comments or other referring
entities? All those comments need to be repointed at the new Photo.
A mess.
-
GOOD IDEA:
class Album {
@Id Long id;
String name;
}
class Photo {
@Id Long id;
OKey album;
String caption;
String blobStoreKey;
}
Fetching photos in an album:
OQuery query = createQuery(Photo.class).filter("album", albumKey);
List fetched = ofy.prepare(query).asList();
You can now move Photos between Albums easily and you can load/query
Albums efficiently.
-
GOOD IDEA #2:
I considered writing something about index entities as described here:
http://code.google.com/events/io/2009/sessions/BuildingScalableComplexApps.html
This is what you would probably want to use if a Photo can live in
more than one Album. But this message is long enough already.
-
So now you're thinking, that's just the representation in the
datastore, wouldn't you rather have a entities that hide all that and
provide an interface like this:
class Album {
@Id Long id;
String name;
List photos;
}
class Photo {
@Id Long id;
String caption;
String blobStoreKey;
}
This is one of those things that looks good in a simple demo but in
the real wo
Re: [appengine-java] Question: best practice to persist medium-large data?
Hi Jeff:
I am here again and have put sometime in Objectify. Thanks for taking
pain at different design patterns. Please let the list known your effort
in the good idea #2, especially in the social graph set intersections
and union.
Duong BaTien
DBGROUPS and BudhNet
On Tue, 2010-01-26 at 14:19 -0800, Jeff Schnitzer wrote:
> On Mon, Jan 25, 2010 at 11:52 PM, John Patterson
> wrote:
> >
> > This is why you configure what type of relationship is used using: @Embed,
> > @Entity(PARENT), @Entity(CHILD) or @Entity(INDEPENDENT)
> > So you have the flexibility to choose configuration _without_ rewriting your
> > code. Very important difference.
>
> The problem is that it *isn't* as simple as just changing an
> annotation. Not in appengine, at any rate. It works in the simple
> case (good for demos and sample apps), but you start to notice edge
> cases:
>
> * In some representations you can add a Photo to an Album in a
> transaction, in some representations you can't.
> * In some representations, multiple queries are required to fetch a
> fetch an Album containing a Photo.
> * Each representation has a completely different query syntax.
>
> Twig exposes the datastore Query object, which means the developer
> gets the full brunt of this exposure.
>
> I don't want to say that it isn't possible to build a system that
> abstracts entities into a sophisticated object graph - clearly, we
> have JDO & JPA. What I'm saying is that the people who created JDO &
> JPA are not idiots (although I do think the creators of JDO's
> annotations are aesthetically challenged). The reason JDO has all
> that complexity and endless configuration and query languages and
> fetch groups and proxies and detaching and whatnot is because that's
> what you need to abstract an arbitrary entity graph.
>
> > Currently the first type of representation is not an option. I do want to
> > add this as it makes very large collections that change often much more
> > efficient. When it is added you could reconfigure your data schema by
> > changing a single annotation. Such a change in Objectify would require the
> > developer to rewrite their entire data layer
>
> You can never just reconfigure your data schema with a single
> annotation, both for the reasons above and because you probably have
> real-world data to migrate. And real-world constraints demand a
> particular schema!
>
> Let's actually answer the original poster's question - how do you
> model a photo album? I hope he's still listening :-)
>
> The first question is how you should model it in the datastore? I'll
> use Objectify's syntax here because it corresponds directly to the
> datastore representation.
>
> Actually, let's start by describing how you SHOULDN'T model a photo album.
>
> -
>
> BAD IDEA #1:
>
> class Album {
> @Id Long id;
> String name;
> List> photos;
> }
> class Photo {
> @Id Long id;
> String caption;
> String blobStoreKey;// key to GAE's blobstore
> }
>
> Fetching photos in an album (again, Objectify syntax but equivalent to
> the datastore operation) is:
>
> List fetched = ofy.get(album.photos);
>
> There are two reasons why this is a bad idea:
>
> 1) You now have a hard limit of 5,000 photos per album, established by GAE.
>
> 2) Every time you load an Album, you must load the entire set of
> Photo keys. Want to generate a list of Album names? You have to load
> all that key data, orders of mangitude more data than what you want.
>
> -
>
> BAD IDEA #2:
>
> class Album {
> @Id Long id;
> String name;
> }
> class Photo {
> @Id Long id;
> @Parent OKey album;
> String caption;
> String blobStoreKey;
> }
>
> This stores the Photo with the Album embedded in the Photo's Key as an
> ancestor, making Photo part of the Album's entity group. At first
> glance, this seems kinda cool and you can now do transactions across
> Albums and Photos.
>
> Fetching photos in an album:
>
> OQuery query = createQuery(Photo.class).ancestor(albumKey);
> List fetched = ofy.prepare(query).asList();
>
> The problem is what happens when you want to move a Photo from one
> Album to another. You can't just change the parent. You must delete
> the Photo entity and create a whole new Photo entity with the new
> parent Album. And if the Photo has Comments or other referring
> entities? All those comments need to be repointed at the new Photo.
> A mess.
>
> -
>
> GOOD IDEA:
>
> class Album {
> @Id Long id;
> String name;
> }
> class Photo {
> @Id Long id;
> OKey album;
> String caption;
> String blobStoreKey;
> }
>
> Fetching photos in an album:
>
> OQuery query = createQuery(Photo.class).filter("album", albumKey);
> List fetched = ofy.prepare(query).asList();
>
> You can now move Photos between Albums easily and you can load/query
> Albums efficiently.
>
> -
>
> GOOD IDEA #2:
>
> I considered writing something about index entities as described here:
>
Re: [appengine-java] Question: best practice to persist medium-large data?
Here's the Objectify version of what's described in the video, capable
of a photo-equivalent of "million user fanout":
class Album {
@Id Long id;
String name;
}
class PhotoIndex {
@Id Long id;
@Parent OKey album;
Set> photos;
}
class Photo {
@Id Long id;
String caption;
String blobStoreKey;
}
If you want to ask "what albums are this photo in?" (equivalent to
"what messages are waiting for me" in the video), you query like this:
OQuery query =
createQuery(PhotoIndex.class).filter("photos", photoKey);
List> keys = ofy.prepareKeysOnly(query).asList();
List albums = ofy.get(keys);
Jeff
On Tue, Jan 26, 2010 at 4:08 PM, Duong BaTien wrote:
> Hi Jeff:
>
> I am here again and have put sometime in Objectify. Thanks for taking
> pain at different design patterns. Please let the list known your effort
> in the good idea #2, especially in the social graph set intersections
> and union.
>
> Duong BaTien
> DBGROUPS and BudhNet
>
>
> On Tue, 2010-01-26 at 14:19 -0800, Jeff Schnitzer wrote:
>> On Mon, Jan 25, 2010 at 11:52 PM, John Patterson
>> wrote:
>> >
>> > This is why you configure what type of relationship is used using: @Embed,
>> > @Entity(PARENT), @Entity(CHILD) or @Entity(INDEPENDENT)
>> > So you have the flexibility to choose configuration _without_ rewriting
>> > your
>> > code. Very important difference.
>>
>> The problem is that it *isn't* as simple as just changing an
>> annotation. Not in appengine, at any rate. It works in the simple
>> case (good for demos and sample apps), but you start to notice edge
>> cases:
>>
>> * In some representations you can add a Photo to an Album in a
>> transaction, in some representations you can't.
>> * In some representations, multiple queries are required to fetch a
>> fetch an Album containing a Photo.
>> * Each representation has a completely different query syntax.
>>
>> Twig exposes the datastore Query object, which means the developer
>> gets the full brunt of this exposure.
>>
>> I don't want to say that it isn't possible to build a system that
>> abstracts entities into a sophisticated object graph - clearly, we
>> have JDO & JPA. What I'm saying is that the people who created JDO &
>> JPA are not idiots (although I do think the creators of JDO's
>> annotations are aesthetically challenged). The reason JDO has all
>> that complexity and endless configuration and query languages and
>> fetch groups and proxies and detaching and whatnot is because that's
>> what you need to abstract an arbitrary entity graph.
>>
>> > Currently the first type of representation is not an option. I do want to
>> > add this as it makes very large collections that change often much more
>> > efficient. When it is added you could reconfigure your data schema by
>> > changing a single annotation. Such a change in Objectify would require the
>> > developer to rewrite their entire data layer
>>
>> You can never just reconfigure your data schema with a single
>> annotation, both for the reasons above and because you probably have
>> real-world data to migrate. And real-world constraints demand a
>> particular schema!
>>
>> Let's actually answer the original poster's question - how do you
>> model a photo album? I hope he's still listening :-)
>>
>> The first question is how you should model it in the datastore? I'll
>> use Objectify's syntax here because it corresponds directly to the
>> datastore representation.
>>
>> Actually, let's start by describing how you SHOULDN'T model a photo album.
>>
>> -
>>
>> BAD IDEA #1:
>>
>> class Album {
>> @Id Long id;
>> String name;
>> List> photos;
>> }
>> class Photo {
>> @Id Long id;
>> String caption;
>> String blobStoreKey; // key to GAE's blobstore
>> }
>>
>> Fetching photos in an album (again, Objectify syntax but equivalent to
>> the datastore operation) is:
>>
>> List fetched = ofy.get(album.photos);
>>
>> There are two reasons why this is a bad idea:
>>
>> 1) You now have a hard limit of 5,000 photos per album, established by GAE.
>>
>> 2) Every time you load an Album, you must load the entire set of
>> Photo keys. Want to generate a list of Album names? You have to load
>> all that key data, orders of mangitude more data than what you want.
>>
>> -
>>
>> BAD IDEA #2:
>>
>> class Album {
>> @Id Long id;
>> String name;
>> }
>> class Photo {
>> @Id Long id;
>> @Parent OKey album;
>> String caption;
>> String blobStoreKey;
>> }
>>
>> This stores the Photo with the Album embedded in the Photo's Key as an
>> ancestor, making Photo part of the Album's entity group. At first
>> glance, this seems kinda cool and you can now do transactions across
>> Albums and Photos.
>>
>> Fetching photos in an album:
>>
>> OQuery query = createQuery(Photo.class).ancestor(albumKey);
>> List fetched = ofy.prepare(query).asList();
>>
>> The problem is what happens when you want to move a Photo from one
>> Album to another. You can't just change the parent. Y
Re: [appengine-java] Question: best practice to persist medium-large data?
On 27 Jan 2010, at 05:19, Jeff Schnitzer wrote: although I do think the creators of JDO's annotations are aesthetically challenged). Ha ha fair enough. Sorry can't respond more just yet as Im out the door to Koh Phangan until Monday! I think its a good thing to have abstractions at different levels - thats why iBatis and Hibernate both thrive in the same domain. Higher level / lower level, more work / less work. Horses for courses. -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To post to this group, send email to google-appengine-j...@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Question: best practice to persist medium-large data?
On Tue, Jan 26, 2010 at 4:40 PM, Jeff Schnitzer wrote:
>
> If you want to ask "what albums are this photo in?" (equivalent to
> "what messages are waiting for me" in the video), you query like this:
>
> OQuery query =
> createQuery(PhotoIndex.class).filter("photos", photoKey);
> List> keys = ofy.prepareKeysOnly(query).asList();
> List albums = ofy.get(keys);
I made a mistake here - you need to grab the parents of the keys. So
the full process is:
OQuery query =
createQuery(PhotoIndex.class).filter("photos", photoKey);
List> albumKeys = new ArrayList>();
for (OKey indexKey: ofy.prepareKeysOnly(query).asIterable())
albumKeys.add(indexKey.getParent());
List albums = ofy.get(albumKeys);
We're making some API changes in v2 which will make this a little prettier.
Jeff
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
Re: [appengine-java] Question: best practice to persist medium-large data?
Thanks Jeff.
There seems to be something at odd:
On Tue, 2010-01-26 at 16:40 -0800, Jeff Schnitzer wrote:
> Here's the Objectify version of what's described in the video, capable
> of a photo-equivalent of "million user fanout":
>
> class Album {
>@Id Long id;
>String name;
> }
> class PhotoIndex {
>@Id Long id;
>@Parent OKey album;
>Set> photos;
> }
> class Photo {
>@Id Long id;
>String caption;
>String blobStoreKey;
> }
>
> If you want to ask "what albums are this photo in?" (equivalent to
> "what messages are waiting for me" in the video), you query like this:
>
> OQuery query =
> createQuery(PhotoIndex.class).filter("photos", photoKey);
> List> keys = ofy.prepareKeysOnly(query).asList();
> List albums = ofy.get(keys);
>
> Jeff
>
In the above query to ask what albums are this photo in, I wonder if it
is legitimate to cast List of PhotoIndex keys with List> so
you can get a particular photoKey in a list of albums.
Using the same structure for User, MessageIndex and Message, the query
seems to ask for a list of users, a particular message is sent to?
class User {
@Id Name userId;
}
class MessageIndex {
@Id Long messageIndexId;
@Parent OKey userKey;
List> messageKeys;
}
class Message {
@Id Long messageId;
String ccontent;
}
To query the list of users a particular messageKey is sent to:
OQuery query =
ObjectifyService.createQuery(MessageIndex.class).filter(messageKeys,
messageKey);
List> userKeys = ofy.prepareKeysOnly(query).asList();
List users = ofy.get(userKeys);
To query what messages are waiting for me:
OQuery query = ObjectifyService.createQuery
(MessageIndex.class).filter(userKey, meKey);
List> listOfMessageKeys = ofy.prepareKeysOnly
(query).asList();
// iterate through listOfMessageKeys and each iteration, get a batch of
messages sent to meKey?
Can GQL of python be used in Objectify?
Thanks
Duong BaTien
DBGROUPS and BudhNet
--
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to google-appengine-j...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengine-java+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en.
