On Dec 23, 2010, at 9:38 PM, Daniel Beatty wrote:
Begin forwarded message:
From: Daniel Beatty <danielbea...@mac.com>
Date: December 23, 2010 6:24:03 PM PST
To: Ramsey Gurley <ram...@xeotech.com>
Cc: Daniel Beatty <danielbea...@mac.com>
Subject: Re: Dynamically changing the number of columns included in
a D2W query/listing
Greetings Ramsey,
Brilliant idea, and it works for the most part. There is another
error that crops up, and I can not seem to make heads or tails of
it. It is a bounds error:
IllegalArgumentException: Index (10) out of bounds [0, 9]
at com.webobjects.foundation.NSArray.objectAtIndex(NSArray.java:379)
at
er
.extensions
.eof
.ERXEOControlUtilities
.ensureSortOrdering(ERXEOControlUtilities.java:2422)
at
er
.extensions
.eof
.ERXEOControlUtilities.objectsInRange(ERXEOControlUtilities.java:617)
at
er
.extensions
.batching
.ERXBatchingDisplayGroup
.objectsInRange(ERXBatchingDisplayGroup.java:368)
at
er
.extensions
.batching
.ERXBatchingDisplayGroup.refetch(ERXBatchingDisplayGroup.java:407)
at
er
.extensions
.batching
.ERXBatchingDisplayGroup.fetch(ERXBatchingDisplayGroup.java:459)
at
er
.directtoweb
.pages.ERD2WListPage._fetchDisplayGroup(ERD2WListPage.java:452)
at er.directtoweb.pages.ERD2WListPage.setupPhase(ERD2WListPage.java:
552)
... skipped 16 stack elements
Any ideas? Weird part is that it happens for some of the bloated
tables, but not all.
Just looking at the code, the end index is produced in the refetch()
method. It might be helpful to set a breakpoint in there to discover
how/when/why it is getting set to 10 instead of 9.
I ran a second test with
160 : *true* => useBatchingDisplayGroup = "true" [BooleanAssignment]
Some of the tables responded correctly, and with little memory
bloat. Others gave me the same Illegal Argument Exception. It
did not crash the WO program, but it does give a page that is
hardly satisfying. In some cases, it gave the Illegal Argument
Exception on the first try, but the second try it performed as
expected. Any ideas?
Later,
Dan Beatty, M.S. CS (B.S. EECS)
Ph.D. Student
Texas Tech University
dan.bea...@mac.com
http://web.me.com/danielbeatty/My_Home_Page/Welcome.html
(806)438-6620
On Dec 22, 2010, at 8:49 AM, Ramsey Gurley wrote:
On Dec 22, 2010, at 11:09 AM, Daniel Beatty wrote:
Thank you for responding Ramsey,
On Dec 22, 2010, at 6:44 AM, Ramsey Gurley wrote:
On Dec 22, 2010, at 1:08 AM, Daniel Beatty wrote:
Greetings Gentlemen, Chuck, and Davids,
I am exploring how to limit the number of columns queried and
used in for a listing in D2W. My experiment uses ERModernLook,
and has a the SDSS DR1 data set. Several of the tables have an
annoyingly large number of columns (641 to be precise) and a
large number of rows (770k of them). If the JVM is tweaked to
3GB of memory, ERModernLook can return a list from a blank
query (taking nearly 2.5GB memory footprint). I have a few
more tables with just about as a bad of a scheme and size
arrangement. It would be nice to use this data set to measure
just how much abuse D2W can handle.
It sounds like you're mainly interested in just thrashing the
app to see what it can do under stress. That database table
sounds sufficient for the job. Are you putting all 770K rows on
a single page? (^_^) I think it would be interesting to see it
using a "real world" setup with a batching display group also.
That would probably reduce the memory usage a lot, but that's
still a ton of columns to render.
For the starting app, I literally reverse engineered the original
database from inside a framework and am driving the program from
an ERModernLook about as straight out of the box (by your example
from WOWODC 2010), and ran it. The next step I took was to
increase the memory available to the JVM to 3GB.
One thing to be proud of, ERModernLook has a semi-batching
display group, but still loads the entire table into memory.
Try
100 : (entity.name = 'BestTSField' and task = 'list') =>
useBatchingDisplayGroup = "true" [BooleanAssignment]
and see if that changes anything. Note from the
ERXBatchingDisplayGroup doc (since the javadocs are down today for
relocation)...
Extends WODisplayGroup in order to provide real batching. This is
done by adding database specific code to the select statement from
the EOFetchSpecification from the WODisplayGroup'sEODataSource
which must be an EODatabaseDataSource. If used with other
datasources, it reverts to the default behaviour.
I am starting with the one of the large tables. Obviously, the
small tables load without flaw. The rule set I have for the
large table (so far) includes the following:
"60 : (entity.name = 'BestTSField' and task = 'list') =>
batchSize = 5 [com.webobjects.directtoweb.Assignment]",
"60 : (entity.name = 'BestTSField' and task = 'list') =>
fetchLimit = 5 [com.webobjects.directtoweb.Assignment]",
"60 : (entity.name = 'BestTSField' and task = 'list') =>
displayPropertyKeys = (run,rerun,camcol,field)
[com.webobjects.directtoweb.Assignment]",
"60 : entity.name = 'BestTSField' => queryDataSourceDelegate =
edu
.ttu.cs.dcglab.dr1.model.ListBestTSFieldQueryDataSourceDelegate
[ERDDelayedObjectCreationAssignment]",
I currently have the last rule disabled since
"edu
.ttu.cs.dcglab.dr1.model.ListBestTSFieldQueryDataSourceDelegate"
is not implemented. While these are set the way they are, it
does not seem to have an effect on retrieval of this large
table. It still basically loads the entire table into memory.
Since this is a "real world" database that even Microsoft
Research tried improve, it seems to be a worthy challenge to make
ERModernLook tackle it. I can podcast the next effort and place
it on scorpius.hpcc.ttu.edu. That machine is a Mac Pro running
OSX Server 10.6.5 (developer seed) with about 6GB or RAM. I
still need to talk the powers that be to open the ports podcast
producer use to allow outside access, but I think that they can
honor a community of interest.
A couple things would be handy for this experiment.
1. How do I use ERDQueryDataSourceDelegateInterface? It
appears that the interface requires implementation of the
queryDataSource and qualifierFromSender methods.
Just the queryDataSource method really. In the documentation I
put on the wiki, I included a qualifierFromSender() method since
customizing the qualifier on a query page seemed to be a common
request.
2. Is there a way to specify different query data sources for
different tasks for the same entity (list versus inspect)?
I think the datasource is passed from the query page to the list
page. I may be mistaken, I'd have to look to be sure. Every
inspect page I've seen just uses the object(). No datasource
involved.
So the task would inherently be "query", it would not matter if
the result were list or inspect.
3. How do I get more run time logging from the WO application
itself? I thought Shark could do something with the JVM. Can
it measure the memory foot print over time? If so, how?
I'm not sure, but I've heard Pascal mention he is using Nagios
to monitor memory usage.
Well lets hope that Pascal has some ideas.
Thank you,
Dan Beatty, M.S. CS (B.S. EECS)
Ph.D. Student
Texas Tech University
dan.bea...@mac.com
http://web.me.com/danielbeatty/My_Home_Page/Welcome.html
(806)438-6620
Ramsey
Thank you,
Dan Beatty, M.S. CS (B.S. EECS)
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/ramsey%40xeotech.com
This email sent to ram...@xeotech.com
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/ramsey%40xeotech.com
This email sent to ram...@xeotech.com
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list (Webobjects-dev@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
