And, to be completely clean, aParent needs to be declared.
On Jan 10, 2008, at 12:05 AM, James C. Lee wrote:
Thought I'd follow up just in case someone needs this in the future
and
searches the archive. Here's what we did (the code in my original
email
missed the dispose()). Thanks for the reminder, Pierre!
public void foo() {
NSArray parents, children;
Enumeration anEnum;
EOEditingContext ecParents, ecChildren;
Parent aParent;
while (fetch more parents) {
ecParents = new EOEditingContext();
// fetch parents with ecParents in 1000 or n increments...
anEnum = parents.objectEnumerator();
while (anEnum.hasMoreElements()) {
aParent = (Parent)anEnum.nextElement();
ecChildren= new EOEditingContext();
children = fetch_with_editingContext(ecChildren);
// processing...
children = null;
ecChildren.dispose();
ecChildren= null;
System.gc();
}
parents = null;
ecParents.dispose();
ecParents = null;
System.gc();
}
}
^James
-----Original Message-----
From: Mr. Pierre Frisch [mailto:[EMAIL PROTECTED]
Sent: Wednesday, January 09, 2008 3:44 PM
To: Drew Thoeni
Cc: Mr. Pierre Frisch; James C. Lee; WebObjects-Dev List
Subject: Re: Large Array Clogging
May I suggest that you call dispose() on the EC when
discarding it. It
will accelerate the GC as this will actively break the object
relations.
Pierre
--
Pierre Frisch
[EMAIL PROTECTED]
On Jan 9, 2008, at 15:04, Drew Thoeni wrote:
James,
Implemented this yesterday and it works as advertised, that is,
perfectly. I can calculate all the stats for the 18,000
parents and
100,000,000 children in an hour too! Thanks for the pattern.
Best,
Drew
On Jan 6, 2008, at 5:55 PM, James C. Lee wrote:
I should clarify that there are separate editing contexts for the
parents
and children:
public void foo() {
NSArray parents;
NSArray children;
Enumeration anEnum;
EOEditingContext ecParent, ecChildren;
ecParent = new EOEditingContext();
// fetch parents with ecParent...
anEnum = parents.objectEnumerator();
while (anEnum.hasMoreElements()) {
aParent = (Parent)anEnum.nextElement();
ecChildren= new EOEditingContext();
children = fetch_with_editingContext(ecChildren);
// processing...
children = null;
ecChildren();
ecChildren= null;
System.gc();
}
}
^James
-----Original Message-----
From:
[EMAIL PROTECTED]
ple.com
[mailto:webobjects-dev-bounces+jcl_applewodev=dreamissary.com@
lists.apple.com] On Behalf Of James C. Lee
Sent: Sunday, January 06, 2008 2:46 PM
To: 'WebObjects-Dev List'
Subject: RE: Large Array Clogging
Drew,
Try this:
public void foo() {
NSArray parents;
NSArray children;
Enumeration anEnum;
EOEditingContext editingContext;
// fetch parents
anEnum = parents.objectEnumerator();
while (anEnum.hasMoreElements()) {
aParent = (Parent)anEnum.nextElement();
editingContext = new EOEditingContext();
children = fetch_with_editingContext(editingContext);
// processing...
children = null;
editingContext.dispose();
editingContext = null;
System.gc();
}
}
^James
-----Original Message-----
From:
[EMAIL PROTECTED]
ple.com
[mailto:webobjects-dev-bounces+jcl_applewodev=dreamissary.com@
lists.apple.com] On Behalf Of Chuck Hill
Sent: Sunday, January 06, 2008 1:57 PM
To: Drew Thoeni
Cc: WebObjects-Dev List
Subject: Re: Large Array Clogging
On Jan 6, 2008, at 11:53 AM, Drew Thoeni wrote:
I have two tables in a parent-child relationship. Some of the
parents have a few children (say a few dozen) and
others have many
children (over 100,000). I'm trying to run through the
parents and
pre-calculate some statistics for their children (for example,
average, mode, standard deviation, etc.).
I'd be tempted to take a long, hard look at some of the
aggregate
functions in Wonder that generate SQL for you. Batch
processing is
not EOF's strongest area due to the overhead of object
creation and
garbage collection.
All this is working except when I reach a specific parent
(#457) the
processing just hangs. I ran this through debug and it
stalls at
the creation of the array for the children.
However, the system would have already processed a similar size
array (#30 has 118,000 children) and it seems it's in a loop.
Activity monitor says the java process is taking 100%.
Other background. I only save the ec after every 20
updates to the
parent records (to reduce write time). And I use
ec.invalidateAllObjects() right after saving to clear
our memory.
Rather than do that, I'd create a new EC and just not
retain _any_
references to the objects previously processed, the previous EC
etc. EOF snapshot counting and Java GC should take care of the
rest.
Finally, there is nothing wrong with the data in
parent/children of
#457 as I can start the process there (or a few parents ahead of
this) and it runs fine.
This seems like a memory problem, but I don't get an out
of memory
exception.
Before OutOfMemory, there is memory starvation. I think that is
what is happening to you. Increase the heap size and see if
that delays
the problem to a later parent. You could also launch with a JVM
parameter of -verbose:gc to log when GC happens.
The solution is to clean up your handling of objects and editing
contexts.
Chuck
--
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve
specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/jcl_appl
ewodev%40dreamissary.com
This email sent to [EMAIL PROTECTED]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/jcl_appl
ewodev%40dreamissary.com
This email sent to [EMAIL PROTECTED]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/drewt7%40comcas
t.net
This email sent to [EMAIL PROTECTED]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/pierre
%40apple.c
om
This email sent to [EMAIL PROTECTED]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/drewt7%40comcast.net
This email sent to [EMAIL PROTECTED]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com
This email sent to [EMAIL PROTECTED]
