I've
tried the solution to have 2 differents mapping files, one that loads only few
objects and the other that loads all the dependent objects.
But I
have some strange problems. Here is the context :
I have
2 JDO objects, one for each database configuration file/mapping
file.
The
first request that I need is a "light" request, so I ask the "light" JDO for the
database object.
The
second request ask all informations => I ask the "full" JDO for the database
object.
when I
execute the second request, Castor use the previously defined database (instead
of the "full" one) and of course the dependent objects are not
loaded.
What's
the problem ? I cannot use two different instance of JDO or Database in the same
thread ?
Did
someone already have this problem ?
Thanks
Ludovic Bertin
-----Message d'origine-----
De : [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Envoyé : jeudi 18 avril 2002 09:26
À : [EMAIL PROTECTED]
Objet : Re: [castor-dev] Performance Issue JDO (repost sorta)I all ready raised the issue, even tried to fix the problem but ended up banging my head against the wall. Correspondence hasnt made it into archieve yet. I have therefore attached the messages below.-----Forwarded by Jan Hansen/Scandinavia/DNK/CARAT on 04/18/2002 09:20AM -----
To: [EMAIL PROTECTED]
From: [EMAIL PROTECTED]
Date: 04/08/2002 03:14PM
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
After using a couple of days at work trying to make required changes to org.exolab.castor.jdo.engine.SQLEngine to solve this issue I have run out of time for now. Boss insists i continue with my normal tasks for now.I have attached the changed functions. Bear in mind that this is 1st attempt to get something working. There are several things I would do better if i had time.1) break dependency on _fields member, this member (and inner class) apears to store redundant information available elsewhere).1.1) this could also make using hashtable for queries un nessesary (this was a really ugly temporary solution).2) clean up error handling (rs.close and stmnt.close called in main body of load, should be called in finally block.3) Transactional safety needs to be adressed. when breaking one big query into several smaller ones something extra needs to be done to assure data intregrity is maintained.Anyway these changes are merely intended to illustrate WHAT needs to be done, not how it SHOULD be done.RegardsJan H. Hansen-----Forwarded by Jan Hansen/Scandinavia/DNK/CARAT on 04/18/2002 09:20AM -----
To: [EMAIL PROTECTED]
From: "Benjamin Voiturier" <[EMAIL PROTECTED]>
Date: 03/15/2002 04:49PM
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
Hello Jan,
I just posted a message mentioning this issue yesterday but did not
receive any feedback yet.
I don't understand why this issue has not been reached before as this is
a critical performance issue that everybody should encounter as soon as
working with an object having several 1:many fields.
Personally I've an object with 5 1:many relations and I'm rapidly
falling in OutOfMemoryException even with a small database (10 objects
in each child means a result set of 100.000 records !!!)
Hope we get some feedback on this.
Regards,
Benjamin.
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: vendredi 15 mars 2002 11:51
To: [EMAIL PROTECTED]
Subject: [castor-dev] Performance Issue JDO (repost sorta)
There seems to be a problem with the way querry builder constructs
querries.
Considder this example:
Instance of Class Parent has
- 1000 Class ChildA instances and
- 1000 Class ChildB instances.
JDO retrieves objects in one querry, the querrys resultset will contain
1 x
1000 x 1000 rows (lazy load or not).
For JDO to "consume" all these rows takes an enourmous amount of
bandwidth,
CPU on database server and CPU on JDO app server.
Alternatively JDO could retrieve objects in 3 querries resulting in 1 +
1000 + 1000 rows.
Is this performance issue something core developers will address ?
If not; Can I make the changes and check them into CVS ?
Regards
Jan H. Hansen
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[EMAIL PROTECTED] with a subject of:
unsubscribe castor-dev
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[EMAIL PROTECTED] with a subject of:
unsubscribe castor-dev
-----Forwarded by Jan Hansen/Scandinavia/DNK/CARAT on 04/18/2002 09:20AM -----
To: [EMAIL PROTECTED]
From: ¿ø Å¿õ <[EMAIL PROTECTED]>
Date: 03/18/2002 11:51AM
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
It seems to me that this is serious problem.
Is somebody working on it?
twwon
-----Original Message-----
From: Benjamin Voiturier [mailto:[EMAIL PROTECTED]]
Sent: Saturday, March 16, 2002 12:49 AM
To: [EMAIL PROTECTED]
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
Hello Jan,
I just posted a message mentioning this issue yesterday but did not receive any feedback yet.
I don't understand why this issue has not been reached before as this is a critical performance issue that everybody should encounter as soon as working with an object having several 1:many fields.
Personally I've an object with 5 1:many relations and I'm rapidly falling in OutOfMemoryException even with a small database (10 objects in each child means a result set of 100.000 records !!!)
Hope we get some feedback on this.
Regards,
Benjamin.
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: vendredi 15 mars 2002 11:51
To: [EMAIL PROTECTED]
Subject: [castor-dev] Performance Issue JDO (repost sorta)
There seems to be a problem with the way querry builder constructs querries.
Considder this example:
Instance of Class Parent has
- 1000 Class ChildA instances and
- 1000 Class ChildB instances.
JDO retrieves objects in one querry, the querrys resultset will contain 1 x 1000 x 1000 rows (lazy load or not).
For JDO to "consume" all these rows takes an enourmous amount of bandwidth, CPU on database server and CPU on JDO app server.
Alternatively JDO could retrieve objects in 3 querries resulting in 1 + 1000 + 1000 rows.
Is this performance issue something core developers will address ?
If not; Can I make the changes and check them into CVS ?
Regards
Jan H. Hansen
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to [EMAIL PROTECTED] with a subject of:
unsubscribe castor-dev
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to [EMAIL PROTECTED] with a subject of:
unsubscribe castor-dev
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[EMAIL PROTECTED] with a subject of:
unsubscribe castor-dev
-----Forwarded by Jan Hansen/Scandinavia/DNK/CARAT on 04/18/2002 09:20AM -----
To: [EMAIL PROTECTED]
From: "Benjamin Voiturier" <[EMAIL PROTECTED]>
Date: 03/26/2002 09:35PM
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
Let's try again...
-----Original Message-----
From: Benjamin Voiturier [mailto:[EMAIL PROTECTED]]
Sent: vendredi 15 mars 2002 16:49
To: '[EMAIL PROTECTED]'
Subject: RE: [castor-dev] Performance Issue JDO (repost sorta)
Hello Jan,
I just posted a message mentioning this issue yesterday but did not
receive any feedback yet.
I don't understand why this issue has not been reached before as this is
a critical performance issue that everybody should encounter as soon as
working with an object having several 1:many fields.
Personally I've an object with 5 1:many relations and I'm rapidly
falling in OutOfMemoryException even with a small database (10 objects
in each child means a result set of 100.000 records !!!)
Hope we get some feedback on this.
Regards,
Benjamin.
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
Sent: vendredi 15 mars 2002 11:51
To: [EMAIL PROTECTED]
Subject: [castor-dev] Performance Issue JDO (repost sorta)
There seems to be a problem with the way querry builder constructs
querries.
Considder this example:
Instance of Class Parent has
- 1000 Class ChildA instances and
- 1000 Class ChildB instances.
JDO retrieves objects in one querry, the querrys resultset will contain
1 x
1000 x 1000 rows (lazy load or not).
For JDO to "consume" all these rows takes an enourmous amount of
bandwidth,
CPU on database server and CPU on JDO app server.
Alternatively JDO could retrieve objects in 3 querries resulting in 1 +
1000 + 1000 rows.
Is this performance issue something core developers will address ?
If not; Can I make the changes and check them into CVS ?
Regards
Jan H. Hansen
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[EMAIL PROTECTED] with a subject of:
unsubscribe castor-dev
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[EMAIL PROTECTED] with a subject of:
unsubscribe castor-dev
-----Forwarded by Jan Hansen/Scandinavia/DNK/CARAT on 04/18/2002 09:20AM -----
To: [EMAIL PROTECTED]
From: Bruce Snyder <[EMAIL PROTECTED]>
Date: 03/27/2002 03:09AM
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
This one time, at band camp, Benjamin Voiturier said:
BV>Personally I've an object with 5 1:many relations and I'm rapidly
BV>falling in OutOfMemoryException even with a small database (10 objects
BV>in each child means a result set of 100.000 records !!!)
BV>
BV>
BV>There seems to be a problem with the way querry builder constructs
BV>querries.
BV>
BV>Considder this example:
BV>Instance of Class Parent has
BV>- 1000 Class ChildA instances and
BV>- 1000 Class ChildB instances.
BV>
BV>JDO retrieves objects in one querry, the querrys resultset will contain
BV>1 x
BV>1000 x 1000 rows (lazy load or not).
BV>
BV>For JDO to "consume" all these rows takes an enourmous amount of
BV>bandwidth,
BV>CPU on database server and CPU on JDO app server.
BV>
BV>Alternatively JDO could retrieve objects in 3 querries resulting in 1 +
BV>1000 + 1000 rows.
BV>
BV>Is this performance issue something core developers will address ?
Jan, Benjamin,
Please post more information about these problems you're experiencing. For
example, please post the mapping descriptor as well as the client code.
Bruce
--
perl -e 'print unpack("u44", "\?0G)U8V4\@4VYY9&5R\"F)R=6-E0&)S;GED97(N;W)G\"\@\`\`");'
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[EMAIL PROTECTED] with a subject of:
unsubscribe castor-dev
-----Forwarded by Jan Hansen/Scandinavia/DNK/CARAT on 04/18/2002 09:20AM -----
To: [EMAIL PROTECTED]
From: "Benjamin Voiturier" <[EMAIL PROTECTED]>
Date: 03/27/2002 05:06PM
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
Hello Bruce,
Attached you will find a small sample illustrating the problem.
The sample works with 4 classes: Parent, ChildA, ChildB and ChildC.
Parent contains three 1:many references to each of the Child classes.
The problem comes from the SELECT clause used by JDO to load the Parent
object (i.e. an object containing several 1:many references to other
classes).
In this case, the SQL statement used to load a Parent object is:
[Castor] SQL for loading test.jdo.Parent: SELECT
Parent.name,ChildA.childID,ChildB.childID,ChildC.childID FROM Parent
LEFT OUTER JOIN ChildA ON Parent.parentID=ChildA.parent LEFT OUTER JOIN
ChildB ON Parent.parentID=ChildB.parent LEFT OUTER JOIN ChildC ON
Parent.parentID=ChildC.parent WHERE Parent.parentID=?
If the database is populated as to have a parent with 100 childA, 100
childB and 100 childC, a select clause like this one returns 1.000.000
rows (100 x 100 x 100). Left outer joins are recursively executed
against the result set of the previous left outer join.
I included the xml mapping file for JDO as well as a Test class you can
run to display SQL statement used by JDO.
Hope you understand what I mean.
Thanks for your feedback.
Benjamin.
-----Original Message-----
From: Bruce Snyder [mailto:[EMAIL PROTECTED]]
Sent: mercredi 27 mars 2002 3:10
To: [EMAIL PROTECTED]
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
This one time, at band camp, Benjamin Voiturier said:
BV>Personally I've an object with 5 1:many relations and I'm rapidly
BV>falling in OutOfMemoryException even with a small database (10
objects
BV>in each child means a result set of 100.000 records !!!)
BV>
BV>
BV>There seems to be a problem with the way querry builder constructs
BV>querries.
BV>
BV>Considder this example:
BV>Instance of Class Parent has
BV>- 1000 Class ChildA instances and
BV>- 1000 Class ChildB instances.
BV>
BV>JDO retrieves objects in one querry, the querrys resultset will
contain
BV>1 x
BV>1000 x 1000 rows (lazy load or not).
BV>
BV>For JDO to "consume" all these rows takes an enourmous amount of
BV>bandwidth,
BV>CPU on database server and CPU on JDO app server.
BV>
BV>Alternatively JDO could retrieve objects in 3 querries resulting in 1
+
BV>1000 + 1000 rows.
BV>
BV>Is this performance issue something core developers will address ?
Jan, Benjamin,
Please post more information about these problems you're experiencing.
For
example, please post the mapping descriptor as well as the client code.
Bruce
--
perl -e 'print unpack("u44",
"\?0G)U8V4\@4VYY9&5R\"F)R=6-E0&)S;GED97(N;W)G\"\@\`\`");'
-----------------------------------------------------------
If you wish to unsubscribe from this mailing, send mail to
[EMAIL PROTECTED] with a subject of:
unsubscribe castor-dev
-----Forwarded by Jan Hansen/Scandinavia/DNK/CARAT on 04/18/2002 09:20AM -----
To: [EMAIL PROTECTED]
From: "Benjamin Voiturier" <[EMAIL PROTECTED]>
Date: 03/28/2002 10:49AM
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
Hi Bruce,
Here is the db dump. In this db Parent contains only 10 childs of each
type which produce a result set of 1000 rows (10 x 10 x 10).
Here is the select statement executed...
SELECT Parent.name,ChildA.childID,ChildB.childID,ChildC.childID
FROM Parent
LEFT OUTER JOIN ChildA ON Parent.parentID=ChildA.parent
LEFT OUTER JOIN ChildB ON Parent.parentID=ChildB.parent
LEFT OUTER JOIN ChildC ON Parent.parentID=ChildC.parent
WHERE Parent.parentID="192168000006101730570950300001"
Best regards,
Benjamin.
-----Original Message-----
From: Bruce Snyder [mailto:[EMAIL PROTECTED]]
Sent: jeudi 28 mars 2002 7:00
To: Benjamin Voiturier
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
This one time, at band camp, Benjamin Voiturier said:
BV>If the database is populated as to have a parent with 100 childA, 100
BV>childB and 100 childC, a select clause like this one returns
1.000.000
BV>rows (100 x 100 x 100). Left outer joins are recursively executed
BV>against the result set of the previous left outer join.
I see that you're using MySQL. Please use mysqldump to dump the data
from your database so that I can run it into one of my databases.
Thanks,
Bruce
--
perl -e 'print unpack("u44",
"\?0G)U8V4\@4VYY9&5R\"F)R=6-E0&)S;GED97(N;W)G\"\@\`\`");'
-----Forwarded by Jan Hansen/Scandinavia/DNK/CARAT on 04/18/2002 09:20AM -----
To: [EMAIL PROTECTED]
From: [EMAIL PROTECTED]
Date: 04/03/2002 02:42PM
Subject: Re: [castor-dev] Performance Issue JDO (repost sorta)
I have been trying to solve this myself, but seem to have bit off more than i can chew.- In buildFinder the ClassDecriptor apears to contain FieldDescriptors that are not instances of JDOFieldDescriptor in some cases (Running test Harness), this makes a bit difficult to build queries since I need the methods of JDOFieldDescriptor.- All that goes on in SQLEngines constructor is a bit hard to follow. Maybe this is only because of limited understanding of castor, but i suspect SQLEngine as a whole could benefit from refactoring.RegardsJan H. Hansen----------------------------------------------------------- If you wish to unsubscribe from this mailing, send mail [EMAIL PROTECTED] with a subject of: unsubscribe castor-dev
