An issue with cache is cache shadowing, the churning as data is copied
from one cache to another to another.
An example is the speed-up achieved on network accesses by using
sendfile or TransmitFile and bypassing up to four levels of buffering
for a message being despatched to a network interface using send or similar.
Another is opening a file using open and writing to it with write and
reading with read when you are transferring buffers full of data to or
from a file. You avoid the extra level of buffer copying inherent in
fopen. If you dispense with the reads and writes and access the virtual
memory directly you get a further win.
A modern OS uses main memory as a file system cache by virtue of its
virtual memory capability. If you take advantage of that your
applications will run faster.
P Kishor wrote:
On 3/22/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
"P Kishor" <[EMAIL PROTECTED]> wrote:
> Richard,
>
> On 3/22/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> ...
> > The problem is that your working set is bigger than your cache
> > which is causing thrashing. I suggest a solution like this:
> >
> > Add entries to table ONE until the table and its unique index get
> > so big that they no longer fit in cache all at once.
>
> What is this "cache" you talk about? Is this the hard disk cache, or
> some kind of OS-level cache, or a SQLite level cache?
All of the above.
> How can I find
> the size of this, and is this cache user-adjustable? If yes, how?
>
On Linux, the "top" command will show you how much OS disk
cache you are currently using. The hard disk cache is usually
small enough that it can be neglected. The OS cache is also
usually much larger than SQLite's own internal cache.
You might run an experiment where you start inserting and
timing each insert, then switch to a different table when
the insert speed drops below a threshold.
ok. I was hoping for something magical like 'showme_cache_size'. This
sounds more voodoo-ish, but do-able. In any case, it doesn't seem that
this OS cache is user-adjustable, at least not without pipe wrenches
and WD-40.
On a note more related to the OP, even if one could adjust the cache
to the working set (and that would be a moving target -- either make
the cache gigantic, or keep on querying the relative sizes of the two
and somehow keep on adjusting the cache, the time taken to determine
uniqueness in a bigger working set will keep on increasing as the
working set itself keeps on increasing... the curve won't be flat.
-----------------------------------------------------------------------------
To unsubscribe, send email to [EMAIL PROTECTED]
-----------------------------------------------------------------------------
