Mark, Let me give you some background first. On any given day I will be reporting on about 80,000 accounts with over 300,000 detail records available for the past 90 days. My clients are using this data to determine employment elgibility, etc. The majority of these clients are individual employers that might have between 1 and 100 accounts. These are no problem. The 'problem' clients are organizations that handle this type of work for multiple employers on a contract basis. These clients can and do have several thousand accounts that could *potentially* have data that needs to be reported.
The approach I took was to write a separate servlet that wakes up every hours, re-sweeps the account code table, re-sweeps the data table, reconciles these into a set of nested maps, and places it into application scope. Part of this process throws out any account that doesn't have data to be reported, otherwise I'd be trying to handle 200,000 accounts. Based on the account and type of data the user can 'see', they are presented a list of accounts where they can pick one, many, or all. The application then accesses the map structure in application scope, extracts the appropriate data based on account, date range, etc., and returns a set of <display /> tables to the user. These table are required to be independantly sortable, pagable, etc. Again, for up to 100 or so accounts, this is extremely fast. It just goes down the toilet once I start getting over about 500 accounts..... Jerry Jalenak Development Manager, Web Publishing LabOne, Inc. 10101 Renner Blvd. Lenexa, KS 66219 (913) 577-1496 [EMAIL PROTECTED] > -----Original Message----- > From: Mark Lowe [mailto:[EMAIL PROTECTED] > Sent: Thursday, March 04, 2004 10:39 AM > To: Struts Users Mailing List > Subject: Re: Best way to handle big search results.. > > > Sound's like you'll need some scrolling mechanism in between this can > range from changing a query string for a jdbc type app or using an > object model like hibernate which supports result scrolling.. I'd say > with reasonable confidence that returning 48,000 records in > one go is a > pretty unreasonable thing to expect. And who's gonna read them all in > one go? :o) > > Lets start at the beginning. How are you retrieving the > records at the > moment? > > On 4 Mar 2004, at 17:20, Jerry Jalenak wrote: > > > In my application I often return large amounts of data - > for instance, > > a > > single user may have access to as many as 8500 accounts; > each account > > may > > have several hundred detail records associated with it. > I've solved my > > initial problem of being able to return thousands of records (for > > instance, > > I can return almost 48,000 detail records in a little under 5 > > seconds), but > > the problem I have now is that it takes FOREVER for the jsp > to render. > > I > > mean, I've waited over an hour for this thing to complete. Does > > anyone have > > any tips on improving / optimizing the performance of a > compiled JSP? > > I'm > > using Tomcat 5.0.18 Stable with J2SDK 1.4.1_02.... > > > > Thanks! > > > > Jerry Jalenak > > Development Manager, Web Publishing > > LabOne, Inc. > > 10101 Renner Blvd. > > Lenexa, KS 66219 > > (913) 577-1496 > > > > [EMAIL PROTECTED] > > > > > >> -----Original Message----- > >> From: Hookom, Jacob [mailto:[EMAIL PROTECTED] > >> Sent: Wednesday, March 03, 2004 1:39 PM > >> To: Struts Users Mailing List > >> Subject: RE: Best way to handle big search results.. > >> > >> > >> We have to work with search results that are sometimes 1000+ > >> items in size. > >> In addition, much of the information we have is not keyed so > >> we cannot say > >> give results between id 2000 and id 20020. > >> > >> Some things I found to help with search results where we did > >> do in memory > >> sorting/paging were: > >> > >> 1. Create view-specific lightweight objects to be pulled > from the DB > >> 2. Memory allowed, it's faster to pull them all at once > than lazy load > >> 4. The real cause of some of our performance problems were > >> the JSP's/HTML in > >> rendering a huge list to the client vs. only showing 20 at once. > >> > >> > >> To handle this, I created a SearchResultListAdaptor that > >> could sit in the > >> session and handle paging (if anyone wants to argue session > >> scope with me, > >> bring it on--). The SearchResultListAdaptor then contained > >> the behavior of > >> sort order/paging, etc. Sorting was done via bean property using a > >> BeanUtilsComparator. So my SearchResultListAdaptor could > >> then work with an > >> list of beans. > >> > >> Best Regards, > >> Jacob > >> > >> -----Original Message----- > >> From: Arne Brutschy [mailto:[EMAIL PROTECTED] > >> Sent: Wednesday, March 03, 2004 12:57 PM > >> To: Struts Users Mailing List > >> Subject: Best way to handle big search results.. > >> > >> Hi, > >> > >> I'm looking for the best way to handle big search results. > >> > >> Setting: the user can search for other users in the ldap > >> directory. This > >> might return a long list of results (~40.000 entrys). > Sadly, OpenLDAP > >> doesn't support server-side sorting, so I have to sort the results > >> myself. I want to display 20 items per page, and the user > can browse > >> through these results. > >> > >> What is the best way to handle these result object? > Sorting once and > >> storing/caching it in the session or searching and sorting > every time > >> the user requests a new page of results? > >> > >> Any thoughts/experiences? > >> > >> Regards, > >> Arne Brutschy > >> > >> > >> > --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: > [EMAIL PROTECTED] > >> > >> > --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: > [EMAIL PROTECTED] > >> > >> > > > > This transmission (and any information attached to it) may be > > confidential and > > is intended solely for the use of the individual or entity > to which it > > is > > addressed. If you are not the intended recipient or the person > > responsible for > > delivering the transmission to the intended recipient, be > advised that > > you > > have received this transmission in error and that any use, > > dissemination, > > forwarding, printing, or copying of this information is strictly > > prohibited. > > If you have received this transmission in error, please immediately > > notify > > LabOne at the following email address: > > [EMAIL PROTECTED] > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > This transmission (and any information attached to it) may be confidential and is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient or the person responsible for delivering the transmission to the intended recipient, be advised that you have received this transmission in error and that any use, dissemination, forwarding, printing, or copying of this information is strictly prohibited. If you have received this transmission in error, please immediately notify LabOne at the following email address: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
