Re: [Components] PersistentObject 1.5 design issues (summary and ideas)
Hi all! I re-wrote the design for PersistentObject 1.5 on basis of the proposed solutions in this thread. The issues are now cleared to my satisfaction. Hope the same for you. Please take a look at it please and comment here. The document can be found as usually in SVN: trunk/PersistentObject/design/design-1.5.txt And rendered as HTML here: http://schlitt.info/ez/persistent_object_1.5.html TIA! Toby -- Mit freundlichen Grüßen / Med vennlig hilsen / With kind regards Tobias Schlitt (GPG: 0xC462BC14) eZ Components Developer [EMAIL PROTECTED] | eZ Systems AS | ez.no -- Components mailing list Components@lists.ez.no http://lists.ez.no/mailman/listinfo/components
Re: [Components] PersistentObject 1.5 design issues (summary and ideas)
Hi Tobias Tobias Schlitt wrote: To summarize the daily usage of the new features in short: Using identity mapping == Change your current code $session = new ezcPersistentSession( ... ); to $innerSession = new ezcPersistentSession( ... ); $session = new ezcPersistentIdentitySession( $innerSession, new ezcPersistentIdentityMap() ); and your existing code will enjoy the flavor of the identity map. No problems here :-) Simple relation prefetching === Change your current code $q = $session-createFindQuery( 'User' ); $q-orderBy( 'name' ); $users = $session-find( $q, 'User' ); foreach ( $users as $user ) { $books = $session-getRelatedObjects( $user, 'Book' ); foreach ( $books as $book ) { // Fetch authors... } } to $q = $session-createFindQuery( 'User' ); $q-orderBy( 'name' ); $users = $session-findWithRelatedObjects( $q, 'User', array( 'Book' = array( 'Author' = true ) ) ); // ... Foreach loops here ... and the related objects are fetched with a join instead of single selects. And where do the Book and Author objects go? I suppose they remain in the identity map until they are fetched. And how do you get them from the identity map? Do you still have to use e.g. $session-getRelatedObjects( $user, 'Book' ); ? I must say that this looks easier than what I could gather from the design docs and all the Set stuff. Another question I have is about performance. All related objects are fetched, so this could potentially get thousands of objects in memory. Have there been any tests (or are they planned) to see how the identity map system copes with loads of objects? It would be nice to get a rough idea about the possible limits, that of course also depend on server configuration. But still, a reference would be nice. Thanks for clearing this up with some code examples! Regards, Hans -- Components mailing list Components@lists.ez.no http://lists.ez.no/mailman/listinfo/components
Re: [Components] PersistentObject 1.5 design issues (summary and ideas)
Tobias Schlitt wrote: The set stuff is only needed, if you intend to fetch just a subset of related obbjects (not all). For example, if you want to fetch the top 3 sold books for each author. It's time the holiday starts... why didn't I make the link? A set is a collection and since the existing methods return everything it only makes sense to add subsets. Quite the Duh! moment now ;-) There is no code, yet, that is why you just find a design document draft, so far. I agree with you, that ending up with thousands of (potentially not used) objects in memory is not a good idea. For this reason, I introduced the set stuff. :) If these features are used in a reasonable way, I expect it to be quite performant. True, any fetching has the potential to be used in a wrong way. If the appropriate methods are there, it's up to the programmer to use them the right way. As the saying goes, there's no point in trying to make something fool proof... Cheers for the explanation, Hans -- Components mailing list Components@lists.ez.no http://lists.ez.no/mailman/listinfo/components
[Components] PersistentObject 1.5 design issues (summary and ideas)
Hi! I thought a bit more about the open issues with the PersistentObject identity map design and try to summarize them (+ suggested solutions) in this mail for further discussions. Refetching objects == Problem --- It might be desirable to re-fetch a set of objects during a request. A global switch in the ezcPersistentIdentityMap instance will allow this to happen. If this switch is turned on, ezcPersistentIdentitySession should fetch the values of the affected objects from the database again and cache them. Solution The $refetch option will be moved from ezcPersistentIdentityMap to ezcPersistentIdentitySession. If it is turned on, the session will not ask the identity map for an identity, but directly issue the query. The results of this query will be replaced into the identity map, using a new replaceObject() method. This method will check, if an identity of the object already exists. Is this the case, it will replace the values in the existing identity with the values of the submitted object and return the existing identity. If no identity exists, the submitted identity will be recorded and returned. The session will always replace the fetched object with the one returned from replaceObject(). Refetching relations Problem --- In PersistentObject 1.5 it shall be possible to pre-fetch related objects. The idea is to provide methods for this pre-fetching and store the fetched objects in the identity map. On calls to getRelatedObjects() the cached results will be returned instead of issueing new SELECT statements. The method ezcPersistentSession-createRelationFindQuery() allows the user to manipulate the query used to fetch related objects (mainly for the WHERE clause). The created query can be used with ezcPersistentSession-find(). If we support relation pre-fetching using the createRelationFindQuery() method, we cannot make sure that the complete set of related objects is fetched, as it would potentially be expected by getRelatedObjects() later. Solution ezcPersistentIdentitySession-createRelationFindQuery() will not support pre-fetching of related objects. It will simple work as it does in ezcPersistentSession and only the normal identity mapping is applied, when using find() to issue the created query object. The $refetchRelations and $cacheRelation options will be removed. To provide pre-fetching of relations, dedicated methods are provided: - findWithRelatedObjects( ezcPersistentFindQuery $q, array $related ) This method will not allow the user to influence the WHERE clause of the relation joins. Only the WHERE clause used to find the source objects can be manipulated. The query $q can be created by ezcPersistentIdentitySession-createFindQuery() or -createRelationFindQuery(). Since the decorated ezcQuerySelect is used here, JOIN operations will be completly forbiddenin this query object to avoid inconsistencies. This behavior ensures that getRelatedObjects() will have the correct relation sets available, whenever it is called. - createdRelationSetFindQuery( string $class, array $relations, string $setName ) This method will return a specially decorated ezcQuerySelect object, that contains some meta information: The classes that will be extracted and the name of the set. The next method described here can be used to fetch this set and store it in a special place in the identity map. Such sets can only be retrieved from ezcPersistentIdentitySession using another special method. - findRelatedObjectSet( ezcPersistentRelationSetQuery $q ) Finds relation sets as defined through the query object build by the createdRelationSetFindQuery() method. The source objects (defined through the $class attribute to createdRelationSetFindQuery()) are fetched as they are and their identity is stored. Assigned to these identities, the related objects (defined through the $relations parameter to createdRelationSetFindQuery()) are not stored in the normal relation map inside the identity map. Instead they are stored in a special relation structure, that reflects the name of the set. The objects identities themselves are created and retrieved from the cache as usual. - getRelatedObjectsSet( object $source, string $class, string $relationName, string $setName ) This method is used to retrieve a related object set from the identity map. It looks up the set defined by the parameters in the structure created by findRelatedObjectSet() and returns it. The solutions above are still rough ideas and need to be defined in more detail in the design document later. However, some comments on the basic ideas would be nice. TIA! Toby -- Mit freundlichen Grüßen / Med vennlig hilsen / With kind regards Tobias Schlitt (GPG: 0xC462BC14) eZ Components Developer [EMAIL PROTECTED] | eZ Systems AS | ez.no -- Components mailing list
Re: [Components] PersistentObject 1.5 design issues (summary and ideas)
Hi! Tobias Schlitt wrot Refetching objects == Problem --- It might be desirable to re-fetch a set of objects during a request. A global switch in the ezcPersistentIdentityMap instance will allow this to happen. If this switch is turned on, ezcPersistentIdentitySession should fetch the values of the affected objects from the database again and cache them. Solution The $refetch option will be moved from ezcPersistentIdentityMap to ezcPersistentIdentitySession. If it is turned on, the session will not ask the identity map for an identity, but directly issue the query. The results of this query will be replaced into the identity map, using a new replaceObject() method. This method will check, if an identity of the object already exists. Is this the case, it will replace the values in the existing identity with the values of the submitted object and return the existing identity. If no identity exists, the submitted identity will be recorded and returned. The session will always replace the fetched object with the one returned from replaceObject(). Is it ok to couple the persistent session with the identity map? I don't mind because i will use it anyway. I agree that it should not be optionnal. Will replaceObject() be used when updating an object? I think it should. Regards, James. -- Components mailing list Components@lists.ez.no http://lists.ez.no/mailman/listinfo/components
Re: [Components] PersistentObject 1.5 design issues (summary and ideas)
Hi! On 07/08/2008 01:02 PM Derick Rethans wrote: On Tue, 8 Jul 2008, Tobias Schlitt wrote: Refetching objects == Problem --- It might be desirable to re-fetch a set of objects during a request. A global switch in the ezcPersistentIdentityMap instance will allow this to happen. If this switch is turned on, ezcPersistentIdentitySession should fetch the values of the affected objects from the database again and cache them. Solution The $refetch option will be moved from ezcPersistentIdentityMap to ezcPersistentIdentitySession. If it is turned on, the session will not ask the identity map for an identity, but directly issue the query. The results of this query will be replaced into the identity map, using a new replaceObject() method. This method will check, if an identity of the object already exists. Is this the case, it will replace the values in the existing identity with the values of the submitted object and return the existing identity. If no identity exists, the submitted identity will be recorded and returned. The session will always replace the fetched object with the one returned from replaceObject(). Isn't this too course-grained? What if I just want to re-fetch for one specific object? Now I'd have turn the option globally on, re-fetch the object and turn it back on. Why can't it be a parameter to load() ? The described case is just relevant if you want to update multiple identities using a find query. In case you only want to update data for 1 object, you'd use ezcPersistentSession-refresh() anyway. This already works as desired. In an open issue in the current design doc I also suggested to enhance the refresh() method to work on an array of objects and perform a single SELECT query to refresh all together. Refetching relations Problem --- In PersistentObject 1.5 it shall be possible to pre-fetch related objects. The idea is to provide methods for this pre-fetching and store the fetched objects in the identity map. On calls to getRelatedObjects() the cached results will be returned instead of issueing new SELECT statements. The method ezcPersistentSession-createRelationFindQuery() allows the user to manipulate the query used to fetch related objects (mainly for the WHERE clause). The created query can be used with ezcPersistentSession-find(). If we support relation pre-fetching using the createRelationFindQuery() method, we cannot make sure that the complete set of related objects is fetched, as it would potentially be expected by getRelatedObjects() later. Solution ezcPersistentIdentitySession-createRelationFindQuery() will not support pre-fetching of related objects. It will simple work as it does in ezcPersistentSession and only the normal identity mapping is applied, when using find() to issue the created query object. The $refetchRelations and $cacheRelation options will be removed. To provide pre-fetching of relations, dedicated methods are provided: - findWithRelatedObjects( ezcPersistentFindQuery $q, array $related ) This method will not allow the user to influence the WHERE clause of the relation joins. Only the WHERE clause used to find the source objects can be manipulated. The query $q can be created by ezcPersistentIdentitySession-createFindQuery() or -createRelationFindQuery(). Since the decorated ezcQuerySelect is used here, JOIN operations will be completly forbiddenin this query object to avoid inconsistencies. WHere/how are you decorating ezcQuerySelect? That was also mentioned earlies in the design discussion. My idea is to provide a wrapper for the ezcQuerySelect, that contains meta info for the persistence session. This would also allow us to deprecate the $class parameter to ezcPersistentSession-find(). It makes sense in several places of the identity map. The decoration will be performed in the create*Query() methods and will be transparent for the user. This behavior ensures that getRelatedObjects() will have the correct relation sets available, whenever it is called. - createdRelationSetFindQuery( string $class, array $relations, string $setName ) This method will return a specially decorated ezcQuerySelect object, that contains some meta information: The classes that will be extracted and the name of the set. The next method described here can be used to fetch this set and store it in a special place in the identity map. Such sets can only be retrieved from ezcPersistentIdentitySession using another special method. - findRelatedObjectSet( ezcPersistentRelationSetQuery $q ) Finds relation sets as defined through the query object build by the createdRelationSetFindQuery() method. The source objects (defined through the $class attribute to createdRelationSetFindQuery()) are fetched as they are and their identity is stored. Assigned to
Re: [Components] PersistentObject 1.5 design issues (summary and ideas)
Derick Rethans wrote: 8-- snip --8 I don't quite see why you need this set stuff... it seems all so complicated. That's my problem with this new proposed stuff for PO. While it sounds nice in theory, for now it looks like an overcomplicated API. Regards, Hans -- Components mailing list Components@lists.ez.no http://lists.ez.no/mailman/listinfo/components
Re: [Components] PersistentObject 1.5 design issues (summary and ideas)
Hi Hans! On 07/08/2008 04:02 PM Hans Melis wrote: Derick Rethans wrote: 8-- snip --8 I don't quite see why you need this set stuff... it seems all so complicated. That's my problem with this new proposed stuff for PO. While it sounds nice in theory, for now it looks like an overcomplicated API. Where exactky do you feel it's overcomplecated? In fact, my recent changes make most things more easy, while it sources out some parts we had in the design all the way into dedicated methods. To summarize the daily usage of the new features in short: Using identity mapping == Change your current code $session = new ezcPersistentSession( ... ); to $innerSession = new ezcPersistentSession( ... ); $session = new ezcPersistentIdentitySession( $innerSession, new ezcPersistentIdentityMap() ); and your existing code will enjoy the flavor of the identity map. Simple relation prefetching === Change your current code $q = $session-createFindQuery( 'User' ); $q-orderBy( 'name' ); $users = $session-find( $q, 'User' ); foreach ( $users as $user ) { $books = $session-getRelatedObjects( $user, 'Book' ); foreach ( $books as $book ) { // Fetch authors... } } to $q = $session-createFindQuery( 'User' ); $q-orderBy( 'name' ); $users = $session-findWithRelatedObjects( $q, 'User', array( 'Book' = array( 'Author' = true ) ) ); // ... Foreach loops here ... and the related objects are fetched with a join instead of single selects. Only if you want to do more complex stuff (like fetching a subset of books for a user instead of all), you need to write a bit more code, which is not that much. Where would you expect easier usage? Thanks for your input, Toby -- Mit freundlichen Grüßen / Med vennlig hilsen / With kind regards Tobias Schlitt (GPG: 0xC462BC14) eZ Components Developer [EMAIL PROTECTED] | eZ Systems AS | ez.no -- Components mailing list Components@lists.ez.no http://lists.ez.no/mailman/listinfo/components
Re: [Components] PersistentObject 1.5 design issues (summary and ideas)
On 08/07/2008, Tobias Schlitt [EMAIL PROTECTED] wrote: Hi Hans! On 07/08/2008 04:02 PM Hans Melis wrote: Derick Rethans wrote: 8-- snip --8 I don't quite see why you need this set stuff... it seems all so complicated. That's my problem with this new proposed stuff for PO. While it sounds nice in theory, for now it looks like an overcomplicated API. Where exactky do you feel it's overcomplecated? I don't think it is the usage of the system that is overcomplicated, it's more the inner workings of it. However, this is a complex matter I remember from our discussions and from what I see you write. I'm not sure if it is actually possible to make this much easier. It makes sense if you think about what is really going on here: a cache of loose objects that can be queried in any way, should be as efficient as possible and that should always be up to date. This is not simple. It does feel a bit tangled up I must admit, many ifs and buts that are not very clear right away. $session = new ezcPersistentIdentitySession( $innerSession, new ezcPersistentIdentityMap() ); What is the reason for exposing the identity map like this? I understand you can roll your own identity map and use it easily this way, but when would you want to do that? Frederik -- Components mailing list Components@lists.ez.no http://lists.ez.no/mailman/listinfo/components