Re: [Components] PersistentObject 1.5 design issues (summary and ideas)

2008-07-14 Thread Tobias Schlitt
Hi all!

I re-wrote the design for PersistentObject 1.5 on basis of the proposed
solutions in this thread. The issues are now cleared to my satisfaction.
Hope the same for you. Please take a look at it please and comment here.

The document can be found as usually in SVN:

trunk/PersistentObject/design/design-1.5.txt

And rendered as HTML here:

http://schlitt.info/ez/persistent_object_1.5.html

TIA!
Toby
-- 
Mit freundlichen Grüßen / Med vennlig hilsen / With kind regards

Tobias Schlitt (GPG: 0xC462BC14) eZ Components Developer

[EMAIL PROTECTED] | eZ Systems AS | ez.no
-- 
Components mailing list
Components@lists.ez.no
http://lists.ez.no/mailman/listinfo/components


Re: [Components] PersistentObject 1.5 design issues (summary and ideas)

2008-07-09 Thread Hans Melis
Hi Tobias

Tobias Schlitt wrote:
 To summarize the daily usage of the new features in short:
 
 Using identity mapping
 ==
 
 Change your current code
 
 $session = new ezcPersistentSession( ... );
 
 to
 
 $innerSession = new ezcPersistentSession( ... );
 $session = new ezcPersistentIdentitySession(
 $innerSession,
 new ezcPersistentIdentityMap()
 );
 
 and your existing code will enjoy the flavor of the identity map.
 

No problems here :-)

 Simple relation prefetching
 ===
 
 Change your current code
 
 $q = $session-createFindQuery( 'User' );
 $q-orderBy( 'name' );
 
 $users = $session-find( $q, 'User' );
 
 foreach ( $users as $user )
 {
 $books = $session-getRelatedObjects( $user, 'Book' );
 foreach ( $books as $book )
 {
  // Fetch authors...
 }
 }
 
 to
 
 $q = $session-createFindQuery( 'User' );
 $q-orderBy( 'name' );
 
 $users = $session-findWithRelatedObjects(
 $q,
 'User',
 array(
 'Book' = array(
 'Author' = true
 )
 )
 );
 
 // ... Foreach loops here ...
 
 
 and the related objects are fetched with a join instead of single selects.
 

And where do the Book and Author objects go? I suppose they remain in 
the identity map until they are fetched. And how do you get them from 
the identity map? Do you still have to use e.g. 
$session-getRelatedObjects( $user, 'Book' ); ?

I must say that this looks easier than what I could gather from the 
design docs and all the Set stuff.

Another question I have is about performance. All related objects are 
fetched, so this could potentially get thousands of objects in memory. 
Have there been any tests (or are they planned) to see how the identity 
map system copes with loads of objects? It would be nice to get a rough 
idea about the possible limits, that of course also depend on server 
configuration. But still, a reference would be nice.

Thanks for clearing this up with some code examples!

Regards,
Hans
-- 
Components mailing list
Components@lists.ez.no
http://lists.ez.no/mailman/listinfo/components


Re: [Components] PersistentObject 1.5 design issues (summary and ideas)

2008-07-09 Thread Hans Melis
Tobias Schlitt wrote:
 The set stuff is only needed, if you intend to fetch just a subset of
 related obbjects (not all). For example, if you want to fetch the top 3
 sold books for each author.
 

It's time the holiday starts... why didn't I make the link? A set is a 
collection and since the existing methods return everything it only 
makes sense to add subsets. Quite the Duh! moment now ;-)

 There is no code, yet, that is why you just find a design document
 draft, so far. I agree with you, that ending up with thousands of
 (potentially not used) objects in memory is not a good idea. For this
 reason, I introduced the set stuff. :)
 
 If these features are used in a reasonable way, I expect it to be quite
 performant.
 

True, any fetching has the potential to be used in a wrong way. If the 
appropriate methods are there, it's up to the programmer to use them the 
right way. As the saying goes, there's no point in trying to make 
something fool proof...

Cheers for the explanation,

Hans
-- 
Components mailing list
Components@lists.ez.no
http://lists.ez.no/mailman/listinfo/components


[Components] PersistentObject 1.5 design issues (summary and ideas)

2008-07-08 Thread Tobias Schlitt
Hi!

I thought a bit more about the open issues with the PersistentObject
identity map design and try to summarize them (+ suggested solutions) in
this mail for further discussions.

Refetching objects
==

Problem
---

It might be desirable to re-fetch a set of objects during a request. A
global switch in the ezcPersistentIdentityMap instance will allow this
to happen. If this switch is turned on, ezcPersistentIdentitySession
should fetch the values of the affected objects from the database again
and cache them.

Solution


The $refetch option will be moved from ezcPersistentIdentityMap to
ezcPersistentIdentitySession. If it is turned on, the session will not
ask the identity map for an identity, but directly issue the query. The
results of this query will be replaced into the identity map, using a
new replaceObject() method. This method will check, if an identity of
the object already exists. Is this the case, it will replace the values
in the existing identity with the values of the submitted object and
return the existing identity. If no identity exists, the submitted
identity will be recorded and returned. The session will always replace
the fetched object with the one returned from replaceObject().


Refetching relations


Problem
---

In PersistentObject 1.5 it shall be possible to pre-fetch related
objects. The idea is to provide methods for this pre-fetching and store
the fetched objects in the identity map. On calls to getRelatedObjects()
the cached results will be returned instead of issueing new SELECT
statements.

The method ezcPersistentSession-createRelationFindQuery() allows the
user to manipulate the query used to fetch related objects (mainly for
the WHERE clause). The created query can be used with
ezcPersistentSession-find(). If we support relation pre-fetching using
the createRelationFindQuery() method, we cannot make sure that the
complete set of related objects is fetched, as it would potentially be
expected by getRelatedObjects() later.

Solution


ezcPersistentIdentitySession-createRelationFindQuery() will not support
pre-fetching of related objects. It will simple work as it does in
ezcPersistentSession and only the normal identity mapping is applied,
when using find() to issue the created query object.

The $refetchRelations and $cacheRelation options will be removed.

To provide pre-fetching of relations, dedicated methods are provided:

- findWithRelatedObjects( ezcPersistentFindQuery $q, array $related )

  This method will not allow the user to influence the WHERE clause of
  the relation joins. Only the WHERE clause used to find the source
  objects can be manipulated.

  The query $q can be created by
  ezcPersistentIdentitySession-createFindQuery() or
  -createRelationFindQuery(). Since the decorated ezcQuerySelect is
  used here, JOIN operations will be completly forbiddenin this query
  object to avoid inconsistencies.

  This behavior ensures that getRelatedObjects() will have the correct
  relation sets available, whenever it is called.

- createdRelationSetFindQuery( string $class, array $relations,
   string $setName )

  This method will return a specially decorated ezcQuerySelect object,
  that contains some meta information: The classes that will be
  extracted and the name of the set. The next method described here can
  be used to fetch this set and store it in a special place in the
  identity map. Such sets can only be retrieved from
  ezcPersistentIdentitySession using another special method.

- findRelatedObjectSet( ezcPersistentRelationSetQuery $q )

  Finds relation sets as defined through the query object build by
  the createdRelationSetFindQuery() method. The source objects (defined
  through the $class attribute to createdRelationSetFindQuery()) are
  fetched as they are and their identity is stored. Assigned to these
  identities, the related objects (defined through the $relations
  parameter to createdRelationSetFindQuery()) are not stored in the
  normal relation map inside the identity map. Instead they are stored
  in a special relation structure, that reflects the name of the set.

  The objects identities themselves are created and retrieved from the
  cache as usual.

- getRelatedObjectsSet( object $source, string $class,
string $relationName, string $setName )

  This method is used to retrieve a related object set from the identity
  map. It looks up the set defined by the parameters in the structure
  created by findRelatedObjectSet() and returns it.

The solutions above are still rough ideas and need to be defined in more
detail in the design document later. However, some comments on the basic
ideas would be nice.

TIA!
Toby
-- 
Mit freundlichen Grüßen / Med vennlig hilsen / With kind regards

Tobias Schlitt (GPG: 0xC462BC14) eZ Components Developer

[EMAIL PROTECTED] | eZ Systems AS | ez.no
-- 
Components mailing list

Re: [Components] PersistentObject 1.5 design issues (summary and ideas)

2008-07-08 Thread James Pic
Hi!

Tobias Schlitt wrot
 Refetching objects
 ==
 
 Problem
 ---
 
 It might be desirable to re-fetch a set of objects during a request. A
 global switch in the ezcPersistentIdentityMap instance will allow this
 to happen. If this switch is turned on, ezcPersistentIdentitySession
 should fetch the values of the affected objects from the database again
 and cache them.
 
 Solution
 
 
 The $refetch option will be moved from ezcPersistentIdentityMap to
 ezcPersistentIdentitySession. If it is turned on, the session will not
 ask the identity map for an identity, but directly issue the query. The
 results of this query will be replaced into the identity map, using a
 new replaceObject() method. This method will check, if an identity of
 the object already exists. Is this the case, it will replace the values
 in the existing identity with the values of the submitted object and
 return the existing identity. If no identity exists, the submitted
 identity will be recorded and returned. The session will always replace
 the fetched object with the one returned from replaceObject().

Is it ok to couple the persistent session with the identity map?
I don't mind because i will use it anyway. I agree that it should not be
optionnal.
Will replaceObject() be used when updating an object? I think it should.

Regards, James.
-- 
Components mailing list
Components@lists.ez.no
http://lists.ez.no/mailman/listinfo/components


Re: [Components] PersistentObject 1.5 design issues (summary and ideas)

2008-07-08 Thread Tobias Schlitt
Hi!

On 07/08/2008 01:02 PM Derick Rethans wrote:
 On Tue, 8 Jul 2008, Tobias Schlitt wrote:

 Refetching objects
 ==

 Problem
 ---

 It might be desirable to re-fetch a set of objects during a request. A
 global switch in the ezcPersistentIdentityMap instance will allow this
 to happen. If this switch is turned on, ezcPersistentIdentitySession
 should fetch the values of the affected objects from the database again
 and cache them.

 Solution
 

 The $refetch option will be moved from ezcPersistentIdentityMap to
 ezcPersistentIdentitySession. If it is turned on, the session will not
 ask the identity map for an identity, but directly issue the query. The
 results of this query will be replaced into the identity map, using a
 new replaceObject() method. This method will check, if an identity of
 the object already exists. Is this the case, it will replace the values
 in the existing identity with the values of the submitted object and
 return the existing identity. If no identity exists, the submitted
 identity will be recorded and returned. The session will always replace
 the fetched object with the one returned from replaceObject().

 Isn't this too course-grained? What if I just want to re-fetch for one 
 specific object? Now I'd have turn the option globally on, re-fetch the 
 object and turn it back on. Why can't it be a parameter to load() ?

The described case is just relevant if you want to update multiple
identities using a find query. In case you only want to update data for
1 object, you'd use ezcPersistentSession-refresh() anyway. This already
works as desired.

In an open issue in the current design doc I also suggested to enhance
the refresh() method to work on an array of objects and perform a single
SELECT query to refresh all together.

 Refetching relations
 

 Problem
 ---

 In PersistentObject 1.5 it shall be possible to pre-fetch related
 objects. The idea is to provide methods for this pre-fetching and store
 the fetched objects in the identity map. On calls to getRelatedObjects()
 the cached results will be returned instead of issueing new SELECT
 statements.

 The method ezcPersistentSession-createRelationFindQuery() allows the
 user to manipulate the query used to fetch related objects (mainly for
 the WHERE clause). The created query can be used with
 ezcPersistentSession-find(). If we support relation pre-fetching using
 the createRelationFindQuery() method, we cannot make sure that the
 complete set of related objects is fetched, as it would potentially be
 expected by getRelatedObjects() later.

 Solution
 

 ezcPersistentIdentitySession-createRelationFindQuery() will not support
 pre-fetching of related objects. It will simple work as it does in
 ezcPersistentSession and only the normal identity mapping is applied,
 when using find() to issue the created query object.

 The $refetchRelations and $cacheRelation options will be removed.

 To provide pre-fetching of relations, dedicated methods are provided:

 - findWithRelatedObjects( ezcPersistentFindQuery $q, array $related )

   This method will not allow the user to influence the WHERE clause of
   the relation joins. Only the WHERE clause used to find the source
   objects can be manipulated.

   The query $q can be created by
   ezcPersistentIdentitySession-createFindQuery() or
   -createRelationFindQuery(). Since the decorated ezcQuerySelect is
   used here, JOIN operations will be completly forbiddenin this query
   object to avoid inconsistencies.

 WHere/how are you decorating ezcQuerySelect?

That was also mentioned earlies in the design discussion.

My idea is to provide a wrapper for the ezcQuerySelect, that contains
meta info for the persistence session. This would also allow us to
deprecate the $class parameter to ezcPersistentSession-find().

It makes sense in several places of the identity map.

The decoration will be performed in the create*Query() methods and will
be transparent for the user.

   This behavior ensures that getRelatedObjects() will have the correct
   relation sets available, whenever it is called.

 - createdRelationSetFindQuery( string $class, array $relations,
string $setName )

   This method will return a specially decorated ezcQuerySelect object,
   that contains some meta information: The classes that will be
   extracted and the name of the set. The next method described here can
   be used to fetch this set and store it in a special place in the
   identity map. Such sets can only be retrieved from
   ezcPersistentIdentitySession using another special method.

 - findRelatedObjectSet( ezcPersistentRelationSetQuery $q )

   Finds relation sets as defined through the query object build by
   the createdRelationSetFindQuery() method. The source objects (defined
   through the $class attribute to createdRelationSetFindQuery()) are
   fetched as they are and their identity is stored. Assigned to 

Re: [Components] PersistentObject 1.5 design issues (summary and ideas)

2008-07-08 Thread Hans Melis
Derick Rethans wrote:
8-- snip --8

 I don't quite see why you need this set stuff... it seems all so 
 complicated.
 

That's my problem with this new proposed stuff for PO. While it sounds 
nice in theory, for now it looks like an overcomplicated API.

Regards,
Hans


-- 
Components mailing list
Components@lists.ez.no
http://lists.ez.no/mailman/listinfo/components


Re: [Components] PersistentObject 1.5 design issues (summary and ideas)

2008-07-08 Thread Tobias Schlitt
Hi Hans!

On 07/08/2008 04:02 PM Hans Melis wrote:
 Derick Rethans wrote:
 8-- snip --8

 I don't quite see why you need this set stuff... it seems all so 
 complicated.

 That's my problem with this new proposed stuff for PO. While it sounds 
 nice in theory, for now it looks like an overcomplicated API.

Where exactky do you feel it's overcomplecated?

In fact, my recent changes make most things more easy, while it sources
out some parts we had in the design all the way into dedicated methods.

To summarize the daily usage of the new features in short:

Using identity mapping
==

Change your current code

$session = new ezcPersistentSession( ... );

to

$innerSession = new ezcPersistentSession( ... );
$session = new ezcPersistentIdentitySession(
$innerSession,
new ezcPersistentIdentityMap()
);

and your existing code will enjoy the flavor of the identity map.

Simple relation prefetching
===

Change your current code

$q = $session-createFindQuery( 'User' );
$q-orderBy( 'name' );

$users = $session-find( $q, 'User' );

foreach ( $users as $user )
{
$books = $session-getRelatedObjects( $user, 'Book' );
foreach ( $books as $book )
{
 // Fetch authors...
}
}

to

$q = $session-createFindQuery( 'User' );
$q-orderBy( 'name' );

$users = $session-findWithRelatedObjects(
$q,
'User',
array(
'Book' = array(
'Author' = true
)
)
);

// ... Foreach loops here ...


and the related objects are fetched with a join instead of single selects.

Only if you want to do more complex stuff (like fetching a subset of
books for a user instead of all), you need to write a bit more code,
which is not that much.

Where would you expect easier usage?

Thanks for your input,
Toby
-- 
Mit freundlichen Grüßen / Med vennlig hilsen / With kind regards

Tobias Schlitt (GPG: 0xC462BC14) eZ Components Developer

[EMAIL PROTECTED] | eZ Systems AS | ez.no
-- 
Components mailing list
Components@lists.ez.no
http://lists.ez.no/mailman/listinfo/components


Re: [Components] PersistentObject 1.5 design issues (summary and ideas)

2008-07-08 Thread Frederik Holljen
On 08/07/2008, Tobias Schlitt [EMAIL PROTECTED] wrote:
 Hi Hans!


  On 07/08/2008 04:02 PM Hans Melis wrote:
   Derick Rethans wrote:
   8-- snip --8

   I don't quite see why you need this set stuff... it seems all so
   complicated.

   That's my problem with this new proposed stuff for PO. While it sounds
   nice in theory, for now it looks like an overcomplicated API.


 Where exactky do you feel it's overcomplecated?
I don't think it is the usage of the system that is overcomplicated,
it's more the inner workings of it. However, this is a complex matter
I remember from our discussions and from what I see you write. I'm not
sure if it is actually possible to make this much easier. It makes
sense if you think about what is really going on here:
a cache of loose objects that can be queried in any way, should be as
efficient as possible and that should always be up to date. 
This is not simple. It does feel a bit tangled up I must admit, many
ifs and buts that are not very clear right away.

$session = new ezcPersistentIdentitySession(
$innerSession,
new ezcPersistentIdentityMap()
 );

What is the reason for exposing the identity map like this? I
understand you can roll your own identity map and use it easily this
way, but when would you want to do that?

Frederik
-- 
Components mailing list
Components@lists.ez.no
http://lists.ez.no/mailman/listinfo/components