Also look at ERXFetchSpecificationBatchIterator. I have not directly
used this one, but the javadoc sounds promising and it was originally
written by Max Muller, so you're pretty safe with it I think :)
/**
* The goal of the fetch specification batch iterator is
* to have the ability to iterate through a fetch specification
* that might fetch one million enterprise objects. Fetching all
* of the objects into a singel editing context is prohibitive
* in the amount of memory needed and in the time taken to
* process all of the rows. The iterator allows one to iterate
* through the fetched objects only hydrating those objects need
* in small bite size pieces. The iterator also allows you to
* swap out editing contexts between calls to <b>nextBatch</b>,
* which will allow the garbage collector to collect the old
* editing context and the previous batch of enterprise objects.
*/
ms
On May 12, 2006, at 4:37 PM, Mike Schrag wrote:
Except that he mentioned it's a many-to-many, so I'm guessing there
is a third entity that is unstated that represents the join class?
If so, then THAT would get faulted as an array (one huge million
element array), but if you traverse through those objects to the
DataRecord (or the DataList the other way, assuming it's not
flattened), you'd get a million individual faults. Death by a
million cuts :)
In one particular case, I have a RawRowList class that supports
prefetching. It wraps my raw rows, implements the java.util.List
interface and hands back real EOs when I need them. It prefetches
ahead of me (when you get an Iterator from it) in blocks of 5 or 10
(or whatever) using. However, this STILL ends up with a big array
of raw rows in memory.
ERXBatchingDisplayGroup will do REAL batching for you. I would
kill your relationship (that is only going to hurt you), make your
own fetch spec, and use ERXBatchingDisplayGroup on it.
ms
On May 12, 2006, at 4:22 PM, Ken Anderson wrote:
Kieran,
First off, I would certainly get rid of the relationship to the
million rows - very nasty stuff. I think I could use a little
more info on exactly what the data records are for and what type
of access pattern you need before I could make any further
recommendations.
Oh - in answer to your question, you will only have a single array
fault for the million rows until it gets triggered. Then, you'll
actually try to load all million rows (so if they actually loaded,
they would be real, not faults).
Ken
On May 12, 2006, at 4:11 PM, Kieran Kelleher wrote:
OK, I've seen how to deal with this in the past somewhere on a
WWDC video or something, but for the life of me I cannot find the
approach right now and I am under the crunch to get this done.
I have DataList objects with up to a million related DataRecord
objects. DataList and DataRecord are related in a many-to-many
join. Usually a DataRecord belongs to no more than 50 DataLists.
So regular dataList.dataRecords() obviously crashes my app when
huge amount of data records in the relationship.
Am I understanding correctly in saying that fetching my one
DataList with a million related DataRecords will create a million
"faults"?
What is the strategy here?
Is it:
- turn off the dataRecords class property visibility (diamond icon)?
- kill the relationship to dataRecords in the DataList entity and
work with qualifiers, fetchspecs and raw rows to grab dataRecords
as needed? How does this affect creating relationships .... just
always add the DataList to the DataRecord's dataLists relationship?
Any best practice tips, pointers or links to same would be
appreciated.
Regards, Kieran
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/lists%
40anderhome.com
This email sent to [EMAIL PROTECTED]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/mschrag%
40mdimension.com
This email sent to [EMAIL PROTECTED]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/mschrag%
40mdimension.com
This email sent to [EMAIL PROTECTED]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com
This email sent to archive@mail-archive.com
