Re: [sqlite] Windows performance problems associated with malloc()
On Fri, Dec 17, 2010 at 7:29 PM, Richard Hipp d...@sqlite.org wrote: For certain full-text search queries against a large database, we are seeing speeds which are 3x faster when using -heap 300M (the memsys5 memory allocator) versus omitting the -heap option and thus using system malloc(). This is on windows7. Similar results are seen with both gcc and vs2010 builds. If you have any large queries that you can run on windows using the command-line shell, I would appreciate you timing those queries using the new shells from the download page, both with -heap 300M and without it, and letting me know about any performance differences you see. Just tested a feature to see the number of memory requests for a given query. It is made by keeping a global variable incremented each time malloc is called (actually it is mapped to a different proc) and showing the difference between the values at the start and at the end of the query. Sqlite version 3.7.4 (fts-enabled) For general queries the number of requests is not great, from 10 to 10,000 and if recalculated to requests per second also not so big (100 per sec for example). But for fts3 a small difference in match operand can lead to huge difference in the number of memory requests and you can see that for the phrase query the request per second value is very big. The queries bellow was made against a db with English wikipedia abstracts. The times measured after several repeats so no disk waiting to be affecting. SELECT Count(*) FROM (SELECT * FROM WikiData WHERE Abstract LIKE %official% LIMIT 500) Result: 500, mem req: 2163 664 ms. SELECT count(docid) FROM WikiFTS3 where Abstract Match 'after before'; Result: 1539, mem requests: 10157 44 ms. SELECT count(docid) FROM WikiFTS3 where Abstract Match 'of the previous' Result: 403, mem requests: 1,627,732 3400 ms I don't know what it means, maybe that some optimization might be possible in the internal implementation of fts3 to minimize the number of memory requests. Thanks Max Vlasov ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
Hi, Windows malloc is slow. There is nothing you can do about it. I have seen huge difference in the same code running on Linux vs Windows (Code which mostly do malloc/realloc/free). I am using google perftools (http://code.google.com/p/google-perftools/) to accelerate malloc. Actually, I am using a patched version which have debug malloc working on win32 too (In addition to being faster) - see details here http://code.google.com/p/google-perftools/issues/detail?id=284 Yoni. On 17/12/2010 6:29 PM, Richard Hipp wrote: An SQLite user has brought to our attention a performance issue in SQLite that seems to be associated with malloc(). If you have insights or corroborating experience with this issue please let me know. SQLite supports a zero-malloc option (see http://www.sqlite.org/malloc.html#memsys5 for details) which uses its own internal memory allocator rather than system malloc(). Earlier today, we patched the command-line shell to allow the zero-malloc option to be turned on. If you do: sqlite3 DATABASE then the regular system memory allocator is used, but if you say: sqlite3 -heap 100M DATABASE then the MEMSYS5 memory allocator will be used with a pool of 100MB of memory to work with. (You can adjust the size of your memory pool for whatever you need.) There are win32 and win64 builds of this updated command-line shell compiled using vs2010 here: http://www.sqlite.org/draft/download.html For certain full-text search queries against a large database, we are seeing speeds which are 3x faster when using -heap 300M (the memsys5 memory allocator) versus omitting the -heap option and thus using system malloc(). This is on windows7. Similar results are seen with both gcc and vs2010 builds. If you have any large queries that you can run on windows using the command-line shell, I would appreciate you timing those queries using the new shells from the download page, both with -heap 300M and without it, and letting me know about any performance differences you see. I also observe that compiling for 64-bit using vs2010 (not an option with my ancient version 2.95.3 gcc cross-compiler) that the queries are an additional 2x faster. I was surprised at the dramatic performance increase in going from 32-bit to 64-bit. Is such a speed-up typical? The use of -heap 300M seems to not make any performance difference on Linux. Any insights into why this is, what we are doing wrong, or what we can do to improve the performance of malloc() on windows will be appreciated. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
This is low probability, but maybe the Fault Tolerant Heap is turned on for sqlite.exe? I believe you'll see the exe mentioned in here if that is the case: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\FTH\State FTH was introduced in Windows 7. Ben ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
I believe the Windows default is to use the LFH on Vista and newer
versions of Windows.
The suggestion by Marcus Grimm to use _set_sbh_threshold() to enable use
of the SBH (small block heap) may help under some usage scenarios on
those platforms.
Just to be precise: The _set_sbh_threshold() setting was not related
to sqlite but to our application in general. A part of the application
used to malloc/free a few thousands small items (a linked pointer list)
and this operation surprisingly dramatically slowed down on XP, not on
Win-7. Recativating the small block heap by using _set_sbh_threshold(512)
did the trick, although it remains strange why it is necessary.
Anyway, Sqlite used to be much more advanced when dealing
with memory and thus it might not be applicable, however fts3
might be another story...
The new exe fixes the stupid dll issue but I'm not able to see
any difference on XP using standard queries; haven't tried fts3 yet.
Marcus
-Shane
On Fri, Dec 17, 2010 at 6:29 PM, Doug pa...@poweradmin.com wrote:
I wonder if HeapSetInformation (which can enable a low-fragmentation
heap)
would be helpful too. You can set it on the process
and the CRT heaps. Note that it's not available in Win2K and earlier.
Doug
-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Marcus Grimm
Sent: Friday, December 17, 2010 9:21 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Windows performance problems associated with
malloc()
An SQLite user has brought to our attention a performance issue in
SQLite that seems to be associated with malloc(). If you have
insights or corroborating experience with this issue please let me
know.
We recently had a malloc/free slowdown issue after changing to VS2008 in
combination with XP.
Not sure if it applies in your case but for us this helps:
--
if( _get_sbh_threshold() 512 )
{
_set_sbh_threshold(512);
}
---
I'm unable to run your sqlite3.exe: MSVCR100.dll no found.
Anyway, maybe the above helps.
KInd regards
Marcus
SQLite supports a zero-malloc option (see
http://www.sqlite.org/malloc.html#memsys5 for details) which uses its
own internal memory allocator rather than system malloc(). Earlier
today, we patched the command-line shell to allow the zero-malloc
option to be turned on. If you do:
sqlite3 DATABASE
then the regular system memory allocator is used, but if you say:
sqlite3 -heap 100M DATABASE
then the MEMSYS5 memory allocator will be used with a pool of 100MB of
memory to work with. (You can adjust the size of your memory pool for
whatever you need.)
There are win32 and win64 builds of this updated command-line shell
compiled using vs2010 here:
http://www.sqlite.org/draft/download.html
For certain full-text search queries against a large database, we are
seeing speeds which are 3x faster when using -heap 300M (the memsys5
memory
allocator) versus omitting the -heap option and thus using system
malloc().
This is on windows7. Similar results are seen with both gcc and
vs2010 builds.
If you have any large queries that you can run on windows using the
command-line shell, I would appreciate you timing those queries using
the new shells from the download page, both with -heap 300M and
without it, and letting me know about any performance differences you
see.
I also observe that compiling for 64-bit using vs2010 (not an option
with my ancient version 2.95.3 gcc cross-compiler) that the queries
are an additional 2x faster. I was surprised at the dramatic
performance increase in going from 32-bit to 64-bit. Is such a
speed-up typical?
The use of -heap 300M seems to not make any performance difference
on Linux.
Any insights into why this is, what we are doing wrong, or what we can
do to improve the performance of malloc() on windows will be
appreciated.
--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
Apparently set_sbh_threshold doesn't work in a DLL.
It uses different heap pointers.
Just beware...
I had found a reference where the guy said to watch out as subtle bugs
were
introduced putting this call inside a DLL.
http://www.kuro5hin.org/story/2001/1/4/173637/9948
Interesting. Thanks for pointing to this.
Please note that this reference is 9 years old, we observed
that problem by moving to VS2008, not in older versions.
Anyway, I agree that this threshold issue is wierd -
Microsoft silently turned it off and claims it is no longer
needed, but fact is that was needed to get the same exe
running smoothly on XP. I guess only Microsoft can tell
what's the story behind... :-)
I would rather not suggest to put something like set_sbh_threshold
in sqlite, my suggestion would be to just compile the sqlite shell
using this call one time for test purpose, see if it changes
the reported malloc issue, and if so: Take it as an further
evidence that the memsys5 which Richard appears to implement
infact is a useful extension to the sqlite core, at least for windows.
Marcus
-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org]
On Behalf Of Marcus Grimm
Sent: Saturday, December 18, 2010 5:31 AM
To: General Discussion of SQLite Database
Subject: EXTERNAL:Re: [sqlite] Windows performance problems associated
with
malloc()
I believe the Windows default is to use the LFH on Vista and newer
versions of Windows.
The suggestion by Marcus Grimm to use _set_sbh_threshold() to enable
use
of the SBH (small block heap) may help under some usage scenarios on
those platforms.
Just to be precise: The _set_sbh_threshold() setting was not related
to sqlite but to our application in general. A part of the application
used to malloc/free a few thousands small items (a linked pointer list)
and this operation surprisingly dramatically slowed down on XP, not on
Win-7. Recativating the small block heap by using _set_sbh_threshold(512)
did the trick, although it remains strange why it is necessary.
Anyway, Sqlite used to be much more advanced when dealing
with memory and thus it might not be applicable, however fts3
might be another story...
The new exe fixes the stupid dll issue but I'm not able to see
any difference on XP using standard queries; haven't tried fts3 yet.
Marcus
-Shane
On Fri, Dec 17, 2010 at 6:29 PM, Doug pa...@poweradmin.com wrote:
I wonder if HeapSetInformation (which can enable a low-fragmentation
heap)
would be helpful too. You can set it on the process
and the CRT heaps. Note that it's not available in Win2K and earlier.
Doug
-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Marcus Grimm
Sent: Friday, December 17, 2010 9:21 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Windows performance problems associated with
malloc()
An SQLite user has brought to our attention a performance issue in
SQLite that seems to be associated with malloc(). If you have
insights or corroborating experience with this issue please let me
know.
We recently had a malloc/free slowdown issue after changing to VS2008
in
combination with XP.
Not sure if it applies in your case but for us this helps:
--
if( _get_sbh_threshold() 512 )
{
_set_sbh_threshold(512);
}
---
I'm unable to run your sqlite3.exe: MSVCR100.dll no found.
Anyway, maybe the above helps.
KInd regards
Marcus
SQLite supports a zero-malloc option (see
http://www.sqlite.org/malloc.html#memsys5 for details) which uses its
own internal memory allocator rather than system malloc(). Earlier
today, we patched the command-line shell to allow the zero-malloc
option to be turned on. If you do:
sqlite3 DATABASE
then the regular system memory allocator is used, but if you say:
sqlite3 -heap 100M DATABASE
then the MEMSYS5 memory allocator will be used with a pool of 100MB of
memory to work with. (You can adjust the size of your memory pool for
whatever you need.)
There are win32 and win64 builds of this updated command-line shell
compiled using vs2010 here:
http://www.sqlite.org/draft/download.html
For certain full-text search queries against a large database, we are
seeing speeds which are 3x faster when using -heap 300M (the memsys5
memory
allocator) versus omitting the -heap option and thus using system
malloc().
This is on windows7. Similar results are seen with both gcc and
vs2010 builds.
If you have any large queries that you can run on windows using the
command-line shell, I would appreciate you timing those queries using
the new shells from the download page, both with -heap 300M and
without it, and letting me know about any performance differences you
see.
I also observe that compiling for 64-bit using vs2010 (not an option
with my ancient version 2.95.3 gcc cross-compiler) that the queries
Re: [sqlite] Windows performance problems associated with malloc()
On Sat, Dec 18, 2010 at 7:29 AM, Marcus Grimm mgr...@medcom-online.de wrote: Interesting. Thanks for pointing to this. Please note that this reference is 9 years old, we observed that problem by moving to VS2008, not in older versions. Anyway, I agree that this threshold issue is wierd - Microsoft silently turned it off and claims it is no longer needed, but fact is that was needed to get the same exe running smoothly on XP. I guess only Microsoft can tell what's the story behind... :-) It's no longer needed because the functionality was integrated with the low-frag heap. It might be work investigating a new allocator for sqlite that calls HeapCreate and related stuff to get its own low-frag heap, however my understanding is that on some versions of Windows there are heuristics to automatically switch over from the default. -- Cory Nelson http://int64.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Windows performance problems associated with malloc()
An SQLite user has brought to our attention a performance issue in SQLite that seems to be associated with malloc(). If you have insights or corroborating experience with this issue please let me know. SQLite supports a zero-malloc option (see http://www.sqlite.org/malloc.html#memsys5 for details) which uses its own internal memory allocator rather than system malloc(). Earlier today, we patched the command-line shell to allow the zero-malloc option to be turned on. If you do: sqlite3 DATABASE then the regular system memory allocator is used, but if you say: sqlite3 -heap 100M DATABASE then the MEMSYS5 memory allocator will be used with a pool of 100MB of memory to work with. (You can adjust the size of your memory pool for whatever you need.) There are win32 and win64 builds of this updated command-line shell compiled using vs2010 here: http://www.sqlite.org/draft/download.html For certain full-text search queries against a large database, we are seeing speeds which are 3x faster when using -heap 300M (the memsys5 memory allocator) versus omitting the -heap option and thus using system malloc(). This is on windows7. Similar results are seen with both gcc and vs2010 builds. If you have any large queries that you can run on windows using the command-line shell, I would appreciate you timing those queries using the new shells from the download page, both with -heap 300M and without it, and letting me know about any performance differences you see. I also observe that compiling for 64-bit using vs2010 (not an option with my ancient version 2.95.3 gcc cross-compiler) that the queries are an additional 2x faster. I was surprised at the dramatic performance increase in going from 32-bit to 64-bit. Is such a speed-up typical? The use of -heap 300M seems to not make any performance difference on Linux. Any insights into why this is, what we are doing wrong, or what we can do to improve the performance of malloc() on windows will be appreciated. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
An SQLite user has brought to our attention a performance issue in SQLite
that seems to be associated with malloc(). If you have insights or
corroborating experience with this issue please let me know.
We recently had a malloc/free slowdown issue after
changing to VS2008 in combination with XP.
Not sure if it applies in your case but for us
this helps:
--
if( _get_sbh_threshold() 512 )
{
_set_sbh_threshold(512);
}
---
I'm unable to run your sqlite3.exe: MSVCR100.dll no found.
Anyway, maybe the above helps.
KInd regards
Marcus
SQLite supports a zero-malloc option (see
http://www.sqlite.org/malloc.html#memsys5 for details) which uses its own
internal memory allocator rather than system malloc(). Earlier today, we
patched the command-line shell to allow the zero-malloc option to be
turned
on. If you do:
sqlite3 DATABASE
then the regular system memory allocator is used, but if you say:
sqlite3 -heap 100M DATABASE
then the MEMSYS5 memory allocator will be used with a pool of 100MB of
memory to work with. (You can adjust the size of your memory pool for
whatever you need.)
There are win32 and win64 builds of this updated command-line shell
compiled
using vs2010 here:
http://www.sqlite.org/draft/download.html
For certain full-text search queries against a large database, we are
seeing
speeds which are 3x faster when using -heap 300M (the memsys5 memory
allocator) versus omitting the -heap option and thus using system
malloc().
This is on windows7. Similar results are seen with both gcc and vs2010
builds.
If you have any large queries that you can run on windows using the
command-line shell, I would appreciate you timing those queries using the
new shells from the download page, both with -heap 300M and without it,
and letting me know about any performance differences you see.
I also observe that compiling for 64-bit using vs2010 (not an option with
my
ancient version 2.95.3 gcc cross-compiler) that the queries are an
additional 2x faster. I was surprised at the dramatic performance
increase
in going from 32-bit to 64-bit. Is such a speed-up typical?
The use of -heap 300M seems to not make any performance difference on
Linux.
Any insights into why this is, what we are doing wrong, or what we can do
to
improve the performance of malloc() on windows will be appreciated.
--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
From: Richard Hipp d...@sqlite.org To: General Discussion of SQLite Database sqlite-users@sqlite.org; sqlite-...@sqlite.org Sent: Friday, December 17, 2010 5:29 PM Subject: [sqlite] Windows performance problems associated with malloc() An SQLite user has brought to our attention a performance issue in SQLite that seems to be associated with malloc(). If you have insights or corroborating experience with this issue please let me know. I have a large Windows application that uses SQLite extensively. Some weeks ago, suddenly began to almost hang in certain operations, where delayed minutes to complete some step that usually take less than a second. I think that the problem was in a malloc operation in a associated library -OpenOffice's Hunspell-. The problem appeared consistently when running in Windows XP, but the same .exe run always perfectly in Windows 7. Some days ago, the problem vanished as suddenly as it appeared, and I suspect that both times was the effect of one of the regular Windows's updates. HTH. A.J.Millan ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
On Fri, Dec 17, 2010 at 8:29 AM, Richard Hipp d...@sqlite.org wrote: I also observe that compiling for 64-bit using vs2010 (not an option with my ancient version 2.95.3 gcc cross-compiler) that the queries are an additional 2x faster. I was surprised at the dramatic performance increase in going from 32-bit to 64-bit. Is such a speed-up typical? Perhaps you've got a lot of register spills, or branch-heavy code that can now use cmov? If you mean 2x while still using the standard malloc() -- in 64-bit Windows, a low-frag heap is used by default. It can give a significant performance boost to allocation-heavy apps. Perhaps you could try enabling it in 32-bit. I generally try to avoid malloc() and use my own pools/arenas, but I'd also be curious to know what pattern is causing malloc() to behave poorly. It's easy to make SQLite CPU-bound, so any efficiency improvement would be a plus. -- Cory Nelson http://int64.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
I'm unable to run your sqlite3.exe: MSVCR100.dll no found. That is visual studio 2010 runtime. John ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
On Fri, Dec 17, 2010 at 2:27 PM, John Drescher dresche...@gmail.com wrote: I'm unable to run your sqlite3.exe: MSVCR100.dll no found. That is visual studio 2010 runtime. Can somebody with windows-foo please explain to me what I need to do to vs2010 so that it generates exe file that doe not depend on non-standard DLLs? John ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
On 12/17/2010 3:12 PM, Richard Hipp wrote: Can somebody with windows-foo please explain to me what I need to do to vs2010 so that it generates exe file that doe not depend on non-standard DLLs? If you are building from the command line, specify /MT (/MTd for debug builds) in place of /MD (/MDd). In the IDE, go to Project | Properties | C/C++ | Code Generation, set Runtime Library to Multithreaded (/MT) or Multithreaded Debug (/MTd), for Release and Debug configuations correspondingly. -- Igor Tandetnik ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
On Fri, Dec 17, 2010 at 3:12 PM, Richard Hipp d...@sqlite.org wrote: On Fri, Dec 17, 2010 at 2:27 PM, John Drescher dresche...@gmail.com wrote: I'm unable to run your sqlite3.exe: MSVCR100.dll no found. That is visual studio 2010 runtime. Can somebody with windows-foo please explain to me what I need to do to vs2010 so that it generates exe file that doe not depend on non-standard DLLs? You probably need to enable multithreaded runtime /MT instead of multi-threaded DLL runtime /MD in the C++ Code Generation tab. Note that I do not have 2010 installed here so I am looking at 2008 menus.. John ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
Quoth Richard Hipp d...@sqlite.org, on 2010-12-17 15:12:58 -0500: Can somebody with windows-foo please explain to me what I need to do to vs2010 so that it generates exe file that doe not depend on non-standard DLLs? AFAIK, the platform-approved solution is to distribute the C runtime library with your application. If you're using Visual Studio, that's from the Visual Studio SDK Redistributable Components packages. This has resulted in a certain outcry on the Web of people trying to figure out how to link with the now-Windows-internal MSVCRT library directly. AIUI it's gotten harder over time. It's not really a supported approach, but articles like [1] insist on figuring out how to do it anyway; you can likely find similar topics elsewhere. Also see [2] from MSDN, particularly section What is the difference between msvcrt.dll and msvcr100.dll?. [1] http://kobyk.wordpress.com/2007/07/20/dynamically-linking-with-msvcrtdll-using-visual-c-2005/ [2] http://msdn.microsoft.com/en-us/library/abx4dbyh.aspx --- Drake Wilson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
I wonder if HeapSetInformation (which can enable a low-fragmentation heap)
would be helpful too. You can set it on the process
and the CRT heaps. Note that it's not available in Win2K and earlier.
Doug
-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Marcus Grimm
Sent: Friday, December 17, 2010 9:21 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Windows performance problems associated with malloc()
An SQLite user has brought to our attention a performance issue in
SQLite that seems to be associated with malloc(). If you have
insights or corroborating experience with this issue please let me know.
We recently had a malloc/free slowdown issue after changing to VS2008 in
combination with XP.
Not sure if it applies in your case but for us this helps:
--
if( _get_sbh_threshold() 512 )
{
_set_sbh_threshold(512);
}
---
I'm unable to run your sqlite3.exe: MSVCR100.dll no found.
Anyway, maybe the above helps.
KInd regards
Marcus
SQLite supports a zero-malloc option (see
http://www.sqlite.org/malloc.html#memsys5 for details) which uses its
own internal memory allocator rather than system malloc(). Earlier
today, we patched the command-line shell to allow the zero-malloc
option to be turned on. If you do:
sqlite3 DATABASE
then the regular system memory allocator is used, but if you say:
sqlite3 -heap 100M DATABASE
then the MEMSYS5 memory allocator will be used with a pool of 100MB of
memory to work with. (You can adjust the size of your memory pool for
whatever you need.)
There are win32 and win64 builds of this updated command-line shell
compiled using vs2010 here:
http://www.sqlite.org/draft/download.html
For certain full-text search queries against a large database, we are
seeing speeds which are 3x faster when using -heap 300M (the memsys5
memory
allocator) versus omitting the -heap option and thus using system
malloc().
This is on windows7. Similar results are seen with both gcc and
vs2010 builds.
If you have any large queries that you can run on windows using the
command-line shell, I would appreciate you timing those queries using
the new shells from the download page, both with -heap 300M and
without it, and letting me know about any performance differences you see.
I also observe that compiling for 64-bit using vs2010 (not an option
with my ancient version 2.95.3 gcc cross-compiler) that the queries
are an additional 2x faster. I was surprised at the dramatic
performance increase in going from 32-bit to 64-bit. Is such a
speed-up typical?
The use of -heap 300M seems to not make any performance difference
on Linux.
Any insights into why this is, what we are doing wrong, or what we can
do to improve the performance of malloc() on windows will be
appreciated.
--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
I believe the Windows default is to use the LFH on Vista and newer
versions of Windows.
The suggestion by Marcus Grimm to use _set_sbh_threshold() to enable use
of the SBH (small block heap) may help under some usage scenarios on
those platforms.
-Shane
On Fri, Dec 17, 2010 at 6:29 PM, Doug pa...@poweradmin.com wrote:
I wonder if HeapSetInformation (which can enable a low-fragmentation heap)
would be helpful too. You can set it on the process
and the CRT heaps. Note that it's not available in Win2K and earlier.
Doug
-Original Message-
From: sqlite-users-boun...@sqlite.org
[mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Marcus Grimm
Sent: Friday, December 17, 2010 9:21 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] Windows performance problems associated with malloc()
An SQLite user has brought to our attention a performance issue in
SQLite that seems to be associated with malloc(). If you have
insights or corroborating experience with this issue please let me know.
We recently had a malloc/free slowdown issue after changing to VS2008 in
combination with XP.
Not sure if it applies in your case but for us this helps:
--
if( _get_sbh_threshold() 512 )
{
_set_sbh_threshold(512);
}
---
I'm unable to run your sqlite3.exe: MSVCR100.dll no found.
Anyway, maybe the above helps.
KInd regards
Marcus
SQLite supports a zero-malloc option (see
http://www.sqlite.org/malloc.html#memsys5 for details) which uses its
own internal memory allocator rather than system malloc(). Earlier
today, we patched the command-line shell to allow the zero-malloc
option to be turned on. If you do:
sqlite3 DATABASE
then the regular system memory allocator is used, but if you say:
sqlite3 -heap 100M DATABASE
then the MEMSYS5 memory allocator will be used with a pool of 100MB of
memory to work with. (You can adjust the size of your memory pool for
whatever you need.)
There are win32 and win64 builds of this updated command-line shell
compiled using vs2010 here:
http://www.sqlite.org/draft/download.html
For certain full-text search queries against a large database, we are
seeing speeds which are 3x faster when using -heap 300M (the memsys5
memory
allocator) versus omitting the -heap option and thus using system
malloc().
This is on windows7. Similar results are seen with both gcc and
vs2010 builds.
If you have any large queries that you can run on windows using the
command-line shell, I would appreciate you timing those queries using
the new shells from the download page, both with -heap 300M and
without it, and letting me know about any performance differences you see.
I also observe that compiling for 64-bit using vs2010 (not an option
with my ancient version 2.95.3 gcc cross-compiler) that the queries
are an additional 2x faster. I was surprised at the dramatic
performance increase in going from 32-bit to 64-bit. Is such a
speed-up typical?
The use of -heap 300M seems to not make any performance difference
on Linux.
Any insights into why this is, what we are doing wrong, or what we can
do to improve the performance of malloc() on windows will be
appreciated.
--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Windows performance problems associated with malloc()
On Fri, Dec 17, 2010 at 12:21 PM, Marcus Grimm mgr...@medcom-online.dewrote: I'm unable to run your sqlite3.exe: MSVCR100.dll no found. I have uploaded new executables, built in such a way (I hope) that no longer requires MSVCR100.dll. Please try again and let me know if the new binaries work. Thanks to all who explained how to set up vs2010 to build without the msvcr100.dll dependency... -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
