Re: [sqlite] can i dynamically move backward and forward in a record-set ?
A very simple and efficient method of holding temporary results is to just read it into a temporary file which is memory mapped. The VM manager takes care of the memory buffering so you do not have to be concerned about upper size limits being constrained by memory resources. The file can be anonymous to make the process tidier (subject to a few limitations). Traversing the list is just a matter of reading forwards and back either by using a file handles and the read API or by using a pointer. It have not tried using a pointer to such a structure in a list control, but it may be possible. Avoiding using mallocs for dynamic memory allocation makes for less possibility of checkerboarding in long running processes. Teg wrote: Hello Roger, The data's from Usenet newsgroups so, they have to know what's there before they can filter down to what they're looking for. Some users will just download all 60,000 items in one go. Some will want to sub-search within the N files listed and some do other things. You know how users are, you plan for something, say no more than 5000 items at a time and they decide they want to see them all. I don't even question it anymore. I just just code it to do what they want. It's so fast because the DB's just used for storage and to resolve dependency's for download. Nothing can touch in memory performance. I've had XOVER data for as many as 60 million posts in memory at a time. Some of the largest groups in Usenet have 60 million or more posts in them. It's tricky keeping memory usage small and still make resolving the information needed to download reasonably fast. Essentially, worker threads waiting on semaphores and queued jobs handle the resolution behind the scenes. The point of the original reply though was, that loading data from DB into another efficient data structure, hash_sets is what I use, performs better than trying to cursor forwards and backwards through the data. It's not always possible but, if the data sizes permit it, that's what I do. I don't shy away from using 100-200 megs of RAM if that's what the user's asking for. C Monday, January 1, 2007, 12:47:45 PM, you wrote: RB -BEGIN PGP SIGNED MESSAGE- RB Hash: SHA1 RB Teg wrote: I do that fairly often in my program, I've found performance wise it's better to just load them up. Even to use a thread to load them and post partial results to the GUI so, the loading's dynamic. 10-60,000 items can be efficiently displayed, at least in Windows if you use a virtual list control. Much over 60,000 items and I see a touch of stalling on column sorts and the like. I've had as many as 200-250,000 items in a list control and it's somewhat slow though still usable. RB I can't imagine any user would scroll through 60,000 items! You are far RB better off listing the first 250 or similar amount and providing a way RB for the user to refine their query, or really request all items. RB Roger RB -BEGIN PGP SIGNATURE- RB Version: GnuPG v1.4.6 (GNU/Linux) RB iD8DBQFFmUlBmOOfHg372QQRAucQAKDIuiHkKcQT/2T1HcKix08dOLNy1wCfQJ5y RB KRy/uIxJ5yI6Fx7d8r//Q+0= RB =JK/+ RB -END PGP SIGNATURE- RB - RB To unsubscribe, send email to [EMAIL PROTECTED] RB - - To unsubscribe, send email to [EMAIL PROTECTED] -
Re[2]: [sqlite] can i dynamically move backward and forward in a record-set ?
Hello John, Thanks. I might look into that. The memory manager I use is a block allocator with sub-allocator and coalesces free'd memory as it's returned to the allocator. I don't actually ever malloc or free. I let STL handle all the memory management so, I don't really have to worry about memory leaks. C Tuesday, January 2, 2007, 8:34:45 AM, you wrote: JS A very simple and efficient method of holding temporary results is to JS just read it into a temporary file which is memory mapped. The VM JS manager takes care of the memory buffering so you do not have to be JS concerned about upper size limits being constrained by memory resources. JS The file can be anonymous to make the process tidier (subject to a few JS limitations). JS Traversing the list is just a matter of reading forwards and back either JS by using a file handles and the read API or by using a pointer. It have JS not tried using a pointer to such a structure in a list control, but it JS may be possible. JS Avoiding using mallocs for dynamic memory allocation makes for less JS possibility of checkerboarding in long running processes. JS Teg wrote: Hello Roger, -- Best regards, Tegmailto:[EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] can i dynamically move backward and forward in a record-set ?
My original use of this method was in a program which would literally run for years. An implementation which would checkerboard the heap would ultimately cause problems despite the absence of leaks. Both Unix and Windows support this POSIX feature, and both permit anonymous allocation so it is a good way of dynamically allocating memory in programs portable between Unix and Windows and for implementing shared memory cacheing, buffering or communication in a portable manner. A simple and high performance way of implementing a modest sized single table is to mmap it as a text file and use a fast string search like Boyer-Moore to locate records (rows) in a table search. It can be used to implement tiny footprint flexible data lookups given reasonable size limits. Teg wrote: Hello John, Thanks. I might look into that. The memory manager I use is a block allocator with sub-allocator and coalesces free'd memory as it's returned to the allocator. I don't actually ever malloc or free. I let STL handle all the memory management so, I don't really have to worry about memory leaks. C Tuesday, January 2, 2007, 8:34:45 AM, you wrote: JS A very simple and efficient method of holding temporary results is to JS just read it into a temporary file which is memory mapped. The VM JS manager takes care of the memory buffering so you do not have to be JS concerned about upper size limits being constrained by memory resources. JS The file can be anonymous to make the process tidier (subject to a few JS limitations). JS Traversing the list is just a matter of reading forwards and back either JS by using a file handles and the read API or by using a pointer. It have JS not tried using a pointer to such a structure in a list control, but it JS may be possible. JS Avoiding using mallocs for dynamic memory allocation makes for less JS possibility of checkerboarding in long running processes. JS Teg wrote: Hello Roger, - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] can i dynamically move backward and forward in a record-set ?
I'm sorry im not fluent in sqlite enough to recommend you anything regarding the subject, however as I read virtual I think you meant DYNAMIC!. Using the right word/s ''sometimes'' helps, with this kind of things though. - Original Message - From: Ohad Eder-Pressman [EMAIL PROTECTED] To: sqlite-users@sqlite.org Sent: Monday, January 01, 2007 11:43 AM Subject: [sqlite] can i dynamically move backward and forward in a record-set ? after a query i get back the result set. it looks like i can only do a 'step' to move one record forward. is there any way i can move back and forward to any index i like ? i am displaying the results of a query that can reach tens of thousands of item. i want to do it all virtual, so that when user jumps with the scroll-bar to item 10,523 i tell the record-set to move to that location. my other option is to cache it myself, but i'll leave that as a last resort. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] can i dynamically move backward and forward in a record-set ?
A step performs a get of the next row. Whether it is next of previous depends upon the SQL. Ohad Eder-Pressman wrote: after a query i get back the result set. it looks like i can only do a 'step' to move one record forward. is there any way i can move back and forward to any index i like ? i am displaying the results of a query that can reach tens of thousands of item. i want to do it all virtual, so that when user jumps with the scroll-bar to item 10,523 i tell the record-set to move to that location. my other option is to cache it myself, but i'll leave that as a last resort. - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] can i dynamically move backward and forward in a record-set ?
Hello Ohad, I do that fairly often in my program, I've found performance wise it's better to just load them up. Even to use a thread to load them and post partial results to the GUI so, the loading's dynamic. 10-60,000 items can be efficiently displayed, at least in Windows if you use a virtual list control. Much over 60,000 items and I see a touch of stalling on column sorts and the like. I've had as many as 200-250,000 items in a list control and it's somewhat slow though still usable. C Monday, January 1, 2007, 9:43:53 AM, you wrote: OEP after a query i get back the result set. OEP it looks like i can only do a 'step' to move one record forward. OEP is there any way i can move back and forward to any index i like ? OEP i am displaying the results of a query that can reach tens of thousands of OEP item. OEP i want to do it all virtual, so that when user jumps with the scroll-bar to OEP item 10,523 i tell the record-set to move to that location. OEP my other option is to cache it myself, but i'll leave that as a last resort. -- Best regards, Tegmailto:[EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] -
Re: [sqlite] can i dynamically move backward and forward in a record-set ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Teg wrote: I do that fairly often in my program, I've found performance wise it's better to just load them up. Even to use a thread to load them and post partial results to the GUI so, the loading's dynamic. 10-60,000 items can be efficiently displayed, at least in Windows if you use a virtual list control. Much over 60,000 items and I see a touch of stalling on column sorts and the like. I've had as many as 200-250,000 items in a list control and it's somewhat slow though still usable. I can't imagine any user would scroll through 60,000 items! You are far better off listing the first 250 or similar amount and providing a way for the user to refine their query, or really request all items. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFFmUlBmOOfHg372QQRAucQAKDIuiHkKcQT/2T1HcKix08dOLNy1wCfQJ5y KRy/uIxJ5yI6Fx7d8r//Q+0= =JK/+ -END PGP SIGNATURE- - To unsubscribe, send email to [EMAIL PROTECTED] -
Re[2]: [sqlite] can i dynamically move backward and forward in a record-set ?
Hello Roger, The data's from Usenet newsgroups so, they have to know what's there before they can filter down to what they're looking for. Some users will just download all 60,000 items in one go. Some will want to sub-search within the N files listed and some do other things. You know how users are, you plan for something, say no more than 5000 items at a time and they decide they want to see them all. I don't even question it anymore. I just just code it to do what they want. It's so fast because the DB's just used for storage and to resolve dependency's for download. Nothing can touch in memory performance. I've had XOVER data for as many as 60 million posts in memory at a time. Some of the largest groups in Usenet have 60 million or more posts in them. It's tricky keeping memory usage small and still make resolving the information needed to download reasonably fast. Essentially, worker threads waiting on semaphores and queued jobs handle the resolution behind the scenes. The point of the original reply though was, that loading data from DB into another efficient data structure, hash_sets is what I use, performs better than trying to cursor forwards and backwards through the data. It's not always possible but, if the data sizes permit it, that's what I do. I don't shy away from using 100-200 megs of RAM if that's what the user's asking for. C Monday, January 1, 2007, 12:47:45 PM, you wrote: RB -BEGIN PGP SIGNED MESSAGE- RB Hash: SHA1 RB Teg wrote: I do that fairly often in my program, I've found performance wise it's better to just load them up. Even to use a thread to load them and post partial results to the GUI so, the loading's dynamic. 10-60,000 items can be efficiently displayed, at least in Windows if you use a virtual list control. Much over 60,000 items and I see a touch of stalling on column sorts and the like. I've had as many as 200-250,000 items in a list control and it's somewhat slow though still usable. RB I can't imagine any user would scroll through 60,000 items! You are far RB better off listing the first 250 or similar amount and providing a way RB for the user to refine their query, or really request all items. RB Roger RB -BEGIN PGP SIGNATURE- RB Version: GnuPG v1.4.6 (GNU/Linux) RB iD8DBQFFmUlBmOOfHg372QQRAucQAKDIuiHkKcQT/2T1HcKix08dOLNy1wCfQJ5y RB KRy/uIxJ5yI6Fx7d8r//Q+0= RB =JK/+ RB -END PGP SIGNATURE- RB - RB To unsubscribe, send email to [EMAIL PROTECTED] RB - -- Best regards, Tegmailto:[EMAIL PROTECTED] - To unsubscribe, send email to [EMAIL PROTECTED] -
