Thank you for your tips on page file settings.
This is going to end up as a commerical product and it can be difficult to
predict what potential users may have and what other programs they might try to
run the same time as mine in the background.
Our application runs up to 20 threads and is pretty I/O intensive, we do advise
them to run it at night or when the system won't be too busy-- but we cannot
force them and I hesitate to attempt to alter their system.
It's a delicate balance, however, when using sqlite we notice memory usage
stays nice and low and processing flows right along. Also, we were using an
in-house, file system based processing history with filenames being the actual
hash codes. It worked great until a user decided he wanted to delete 2.8
million items. That took many, many hours and other databases such as Microsoft
Access were too bulky for internet download/setup.
Hello SQLITE. :-)
Thanks for your memory and page file advice. That is what is going on with the
hash tables.
Tom
----------------------------------------
From: Teg <[EMAIL PROTECTED]>
Sent: Monday, December 19, 2005 1:24 PM
To: "[EMAIL PROTECTED]" <sqlite-users@sqlite.org>
Subject: Re[2]: [sqlite] multiple Db's and journal file time hit?
Hello sales,
What are you using for a data structure? I regularly have literally
millions of hashed data structures in memory without encountering this.
If it's windows, I know exactly what it is though, windows pages
things out on a whim so, if you're not constantly touching your data
structures, they get paged out. It's easy to tell if you've been
paged, turn on the VM column in task manager and compare memory usage
to virtual memory size. If there's a large difference then there you
go.
One solution I've used, is to simply disable the page file and add
more RAM. In that way windows can't page your app out. With 2G of
memory and no page file, I never run out even when running games.
I use "hash_sets" for my hashed data structures. I use the one that
comes with STLPort. I'm not suggesting not using SQLite, I am
suggesting that 250K hashes isn't really that many.
C
Monday, December 19, 2005, 4:02:24 PM, you wrote:
stc> Yes the timestamp method is one we have done from within
stc> our own code and it may very well be the operating system
stc> catching up to where it should be and not the database's fault at
stc> all.
stc> It hangs on a read after the inserts--- ourDB.Exec("Select
stc> x From Y where x.HashCode = z LIMIT 1;").
stc> (where x.HashCode is unique.)
stc> Originally we were just storing all hash codes in memory
stc> but on our text box (512MB RAM/4 GIG VRam/40G HD) after about
stc> 250,000 hashcodes (32 bit signed integers) we encountered wicked
stc> memory thrashing (physical / virtual) to the point where the
stc> system would (appear to) lock up and had to abandon the idea for
stc> the database.
stc> It eventually does complete, the first time it takes about
stc> 30 seconds and does not generate any errors. After that one time
stc> we can hit the database 10-20 times a second with no problems for
stc> over 12 hours (length of our test).
stc> Again it may very well be the operating system catching up
stc> to where it should be and not the database's fault at all.
stc> Thanks for all your good advice.
stc> Tom
stc> ----------------------------------------
stc> From: Dennis Jenkins
stc> Sent: Monday, December 19, 2005 11:25 AM
stc> To: sqlite-users@sqlite.org
stc> Subject: Re: [sqlite] multiple Db's and journal file time hit?
stc> [EMAIL PROTECTED] wrote:
>>The delay seems to coincide with the journal file creation-- it
>>happens after our first (committed but not yet written to disk)
>>write attempt to the database, the journal file does not at-the-time
>>exist, and there are 500+ inserts pending with reads rapidly being
>>added after that.
>>
>>Although I too have seen anti-virus programs hose things up, we
>>don't have any running. It was a good thing to check for though.
>>
>>We can live with this minor start-up penalty because our users
>>will oftentimes run this app 12-36 hours straight and could end up
>>hitting the database 1,000,000 times in that period.
>>
>>It does not appear to be any kind of error as no exceptions are
>>thrown. We can create a minor user-appreciated perception of
>>busyness on-screen. :-)
>>
>>Tom
>>
>>----------------------------------------
>> From: [EMAIL PROTECTED]
>>Sent: Monday, December 19, 2005 8:59 AM
>>To: sqlite-users@sqlite.org
>>Subject: Re: [sqlite] multiple Db's and journal file time hit?
>>
>>[EMAIL PROTECTED] wrote:
>>
>>
>>>We just noticed a 30-40 second hit at early on in our program running. >
>>>
>>>
>>
>>Others have reported things like this caused by anti-virus
>>software running on the same machine and insisting on doing
>>some kind of virus scan the first time the journal file is
>>created.
>>--
>>D. Richard Hipp
>>
>>
stc> A few suggestions:
stc> White box it:
stc> Compile SQLITE yourself with debugging symbols. Run your code in a
stc> profiler. Then run it again in a debugger examine the regions of code
stc> that the profiler flagged as "taking a long time".
stc> Black box it:
stc> Use "filemon" (win32/64, from www.sysinternals.com) or "strace" on Linux
stc> or the equivilent on any other platform. They will timestamp and record
stc> all disk access. Have your own program display the exact system
stc> timestamps before and after the lengthy operation. Examine the logs to
stc> see what is going on.
--
Best regards,
Teg mailto:[EMAIL PROTECTED]
