Hello Roger, The data's from Usenet newsgroups so, they have to know what's there before they can filter down to what they're looking for. Some users will just download all 60,000 items in one go. Some will want to sub-search within the N files listed and some do other things. You know how users are, you plan for something, say no more than 5000 items at a time and they decide they want to see them all. I don't even question it anymore. I just just code it to do what they want.
It's so fast because the DB's just used for storage and to resolve dependency's for download. Nothing can touch in memory performance. I've had XOVER data for as many as 60 million posts in memory at a time. Some of the largest groups in Usenet have 60 million or more posts in them. It's tricky keeping memory usage small and still make resolving the information needed to download reasonably fast. Essentially, worker threads waiting on semaphores and queued jobs handle the resolution behind the scenes. The point of the original reply though was, that loading data from DB into another efficient data structure, hash_sets is what I use, performs better than trying to cursor forwards and backwards through the data. It's not always possible but, if the data sizes permit it, that's what I do. I don't shy away from using 100-200 megs of RAM if that's what the user's asking for. C Monday, January 1, 2007, 12:47:45 PM, you wrote: RB> -----BEGIN PGP SIGNED MESSAGE----- RB> Hash: SHA1 RB> Teg wrote: >> I do that fairly often in my program, I've found performance wise it's >> better to just load them up. Even to use a thread to load them and >> post partial results to the GUI so, the loading's dynamic. 10-60,000 >> items can be efficiently displayed, at least in Windows if you use a >> virtual list control. Much over 60,000 items and I see a touch of >> stalling on column sorts and the like. I've had as many as 200-250,000 >> items in a list control and it's somewhat slow though still usable. RB> I can't imagine any user would scroll through 60,000 items! You are far RB> better off listing the first 250 or similar amount and providing a way RB> for the user to refine their query, or really request all items. RB> Roger RB> -----BEGIN PGP SIGNATURE----- RB> Version: GnuPG v1.4.6 (GNU/Linux) RB> iD8DBQFFmUlBmOOfHg372QQRAucQAKDIuiHkKcQT/2T1HcKix08dOLNy1wCfQJ5y RB> KRy/uIxJ5yI6Fx7d8r//Q+0= RB> =JK/+ RB> -----END PGP SIGNATURE----- RB> ----------------------------------------------------------------------------- RB> To unsubscribe, send email to [EMAIL PROTECTED] RB> ----------------------------------------------------------------------------- -- Best regards, Teg mailto:[EMAIL PROTECTED] ----------------------------------------------------------------------------- To unsubscribe, send email to [EMAIL PROTECTED] -----------------------------------------------------------------------------
