A very simple and efficient method of holding temporary results is to
just read it into a temporary file which is memory mapped. The VM
manager takes care of the memory buffering so you do not have to be
concerned about upper size limits being constrained by memory resources.
The file can be anonymous to make the process tidier (subject to a few
limitations).
Traversing the list is just a matter of reading forwards and back either
by using a file handles and the read API or by using a pointer. It have
not tried using a pointer to such a structure in a list control, but it
may be possible.
Avoiding using mallocs for dynamic memory allocation makes for less
possibility of checkerboarding in long running processes.
Teg wrote:
Hello Roger,
The data's from Usenet newsgroups so, they have to know what's there
before they can filter down to what they're looking for. Some users
will just download all 60,000 items in one go. Some will want to
sub-search within the N files listed and some do other things. You
know how users are, you plan for something, say no more than 5000
items at a time and they decide they want to see them all. I don't
even question it anymore. I just just code it to do what they want.
It's so fast because the DB's just used for storage and to resolve
dependency's for download. Nothing can touch in memory performance.
I've had XOVER data for as many as 60 million posts in memory at a
time. Some of the largest groups in Usenet have 60 million or more
posts in them. It's tricky keeping memory usage small and still make
resolving the information needed to download reasonably fast.
Essentially, worker threads waiting on semaphores and queued jobs
handle the resolution behind the scenes.
The point of the original reply though was, that loading data from DB
into another efficient data structure, hash_sets is what I use,
performs better than trying to cursor forwards and backwards through
the data. It's not always possible but, if the data sizes permit it,
that's what I do.
I don't shy away from using 100-200 megs of RAM if that's what the
user's asking for.
C
Monday, January 1, 2007, 12:47:45 PM, you wrote:
RB> -----BEGIN PGP SIGNED MESSAGE-----
RB> Hash: SHA1
RB> Teg wrote:
I do that fairly often in my program, I've found performance wise it's
better to just load them up. Even to use a thread to load them and
post partial results to the GUI so, the loading's dynamic. 10-60,000
items can be efficiently displayed, at least in Windows if you use a
virtual list control. Much over 60,000 items and I see a touch of
stalling on column sorts and the like. I've had as many as 200-250,000
items in a list control and it's somewhat slow though still usable.
RB> I can't imagine any user would scroll through 60,000 items! You are far
RB> better off listing the first 250 or similar amount and providing a way
RB> for the user to refine their query, or really request all items.
RB> Roger
RB> -----BEGIN PGP SIGNATURE-----
RB> Version: GnuPG v1.4.6 (GNU/Linux)
RB> iD8DBQFFmUlBmOOfHg372QQRAucQAKDIuiHkKcQT/2T1HcKix08dOLNy1wCfQJ5y
RB> KRy/uIxJ5yI6Fx7d8r//Q+0=
RB> =JK/+
RB> -----END PGP SIGNATURE-----
RB>
-----------------------------------------------------------------------------
RB> To unsubscribe, send email to [EMAIL PROTECTED]
RB>
-----------------------------------------------------------------------------
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
