> Apparently set_sbh_threshold doesn't work in a DLL. > It uses different heap pointers. > Just beware... > I had found a reference where the guy said to watch out as subtle bugs > were > introduced putting this call inside a DLL. > http://www.kuro5hin.org/story/2001/1/4/173637/9948
Interesting. Thanks for pointing to this. Please note that this reference is 9 years old, we observed that problem by moving to VS2008, not in older versions. Anyway, I agree that this threshold issue is wierd - Microsoft silently turned it off and claims it is no longer needed, but fact is that was needed to get the same exe running smoothly on XP. I guess only Microsoft can tell what's the story behind... :-) I would rather not suggest to put something like set_sbh_threshold in sqlite, my suggestion would be to just compile the sqlite shell using this call one time for test purpose, see if it changes the reported malloc issue, and if so: Take it as an further evidence that the memsys5 which Richard appears to implement infact is a useful extension to the sqlite core, at least for windows. Marcus > > > > -----Original Message----- > From: sqlite-users-boun...@sqlite.org > [mailto:sqlite-users-boun...@sqlite.org] > On Behalf Of Marcus Grimm > Sent: Saturday, December 18, 2010 5:31 AM > To: General Discussion of SQLite Database > Subject: EXTERNAL:Re: [sqlite] Windows performance problems associated > with > malloc() > >> I believe the Windows default is to use the LFH on Vista and newer >> versions of Windows. >> The suggestion by Marcus Grimm to use _set_sbh_threshold() to enable >> use >> of the SBH (small block heap) may help under some usage scenarios on >> those platforms. > > Just to be precise: The _set_sbh_threshold() setting was not related > to sqlite but to our application in general. A part of the application > used to malloc/free a few thousands small items (a linked pointer list) > and this operation surprisingly dramatically slowed down on XP, not on > Win-7. Recativating the small block heap by using _set_sbh_threshold(512) > did the trick, although it remains strange why it is necessary. > Anyway, Sqlite used to be much more advanced when dealing > with memory and thus it might not be applicable, however fts3 > might be another story... > > The new exe fixes the stupid dll issue but I'm not able to see > any difference on XP using standard queries; haven't tried fts3 yet. > > Marcus > > > >> >> -Shane >> >> >> On Fri, Dec 17, 2010 at 6:29 PM, Doug <pa...@poweradmin.com> wrote: >>> I wonder if HeapSetInformation (which can enable a low-fragmentation >>> heap) >>> would be helpful too. You can set it on the process >>> and the CRT heaps. Note that it's not available in Win2K and earlier. >>> >>> Doug >>> >>> -----Original Message----- >>> From: sqlite-users-boun...@sqlite.org >>> [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Marcus Grimm >>> Sent: Friday, December 17, 2010 9:21 AM >>> To: General Discussion of SQLite Database >>> Subject: Re: [sqlite] Windows performance problems associated with >>> malloc() >>> >>>> An SQLite user has brought to our attention a performance issue in >>>> SQLite that seems to be associated with malloc(). If you have >>>> insights or corroborating experience with this issue please let me >>>> know. >>> >>> We recently had a malloc/free slowdown issue after changing to VS2008 >>> in >>> combination with XP. >>> Not sure if it applies in your case but for us this helps: >>> -- >>> if( _get_sbh_threshold() < 512 ) >>> { >>> _set_sbh_threshold(512); >>> } >>> --- >>> >>> I'm unable to run your sqlite3.exe: MSVCR100.dll no found. >>> >>> Anyway, maybe the above helps. >>> >>> KInd regards >>> >>> Marcus >>> >>>> >>>> SQLite supports a "zero-malloc option" (see >>>> http://www.sqlite.org/malloc.html#memsys5 for details) which uses its >>>> own internal memory allocator rather than system malloc(). Earlier >>>> today, we patched the command-line shell to allow the zero-malloc >>>> option to be turned on. If you do: >>>> >>>> sqlite3 DATABASE >>>> >>>> then the regular system memory allocator is used, but if you say: >>>> >>>> sqlite3 -heap 100M DATABASE >>>> >>>> then the MEMSYS5 memory allocator will be used with a pool of 100MB of >>>> memory to work with. (You can adjust the size of your memory pool for >>>> whatever you need.) >>>> >>>> There are win32 and win64 builds of this updated command-line shell >>>> compiled using vs2010 here: >>>> >>>> http://www.sqlite.org/draft/download.html >>>> >>>> For certain full-text search queries against a large database, we are >>>> seeing speeds which are 3x faster when using "-heap 300M" (the memsys5 >>>> memory >>>> allocator) versus omitting the -heap option and thus using system >>>> malloc(). >>>> This is on windows7. Similar results are seen with both gcc and >>>> vs2010 builds. >>>> >>>> If you have any large queries that you can run on windows using the >>>> command-line shell, I would appreciate you timing those queries using >>>> the new shells from the download page, both with "-heap 300M" and >>>> without it, and letting me know about any performance differences you >>>> see. >>>> >>>> I also observe that compiling for 64-bit using vs2010 (not an option >>>> with my ancient version 2.95.3 gcc cross-compiler) that the queries >>>> are an additional 2x faster. I was surprised at the dramatic >>>> performance increase in going from 32-bit to 64-bit. Is such a >>>> speed-up typical? >>>> >>>> The use of "-heap 300M" seems to not make any performance difference >>>> on Linux. >>>> >>>> Any insights into why this is, what we are doing wrong, or what we can >>>> do to improve the performance of malloc() on windows will be >>>> appreciated. >>>> >>>> -- >>>> D. Richard Hipp >>>> d...@sqlite.org >>>> _______________________________________________ >>>> sqlite-users mailing list >>>> sqlite-users@sqlite.org >>>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >>>> >>> >>> >>> _______________________________________________ >>> sqlite-users mailing list >>> sqlite-users@sqlite.org >>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >>> >>> >>> _______________________________________________ >>> sqlite-users mailing list >>> sqlite-users@sqlite.org >>> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >>> >> _______________________________________________ >> sqlite-users mailing list >> sqlite-users@sqlite.org >> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users >> > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
