, particularly before 9am. Has
anybody played with mlock and/or madvise within the sqlite mmap subsystem to
improve this behaviour?
Is this a genuine Linux machine running on physical hardware, or is it a
virtual machine ?
Yes, it's a genuine physical, we have Xeon and Epyc CPUs available.
Some
On Aug 3, 2018, at 1:36 PM, Shevek wrote:
>
> the database is read-only
In that case, I’d just create a :memory: DB on application startup, attach to
the disk copy, use the INSERT FROM … SELECT pattern [1] to clone the data
content within a single transaction, create the indexes, and detach fr
ybody played with mlock and/or madvise within the sqlite mmap subsystem to
> improve this behaviour?
Is this a genuine Linux machine running on physical hardware, or is it a
virtual machine ?
Are you intentionally doing anything that would contend for this memory ? In
other words, when
sqlite mmap subsystem to improve this behaviour?
The system has a few hundred gig of RAM, no swap, the database is
read-only, and we would prefer a page-out to a process crash, so mlock
might not be ideal, but madvise might not be strong enough?
Thank you.
S
Roger Binns wrote:
> It would also be really nice if there wasn't a 2GB mmap limit on 64
> bit machines. The database would fit in my RAM around 4 times, and in
> the address space more times than there are grains of sand! Yea I
> know this isn't very Lite ...
SQLightning has no such limit... ht
On 08/07/2015 12:35 AM, Roger Binns wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 08/06/2015 09:27 AM, Dan Kennedy wrote:
>>> Is it using more CPU cycles in mmap mode or just taking longer?
>>> If the former, does [time] attribute them to "user" or "system"?
> It is taking longer.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 08/07/2015 02:55 AM, Dan Kennedy wrote:
>> http://www.sqlite.org/src/info/3a82c8e6cb7227fe
>
>> Does that improve performance any in your case?
I tested 3a82c8e6 (fix above) against e596a6b6 (previous changeset),
in both cases starting with an emp
On 08/06/2015 09:53 PM, Roger Binns wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> In my testing I am finding that using SQLite's mmap support with
> larger databases (final db size is ~10GB) to create the database (WAL
> mode) is considerably (about three times) slower than no mmap.
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 08/06/2015 09:27 AM, Dan Kennedy wrote:
>> Is it using more CPU cycles in mmap mode or just taking longer?
>> If the former, does [time] attribute them to "user" or "system"?
It is taking longer. I have 3 XML dumps which I turn into JSON
(incremen
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
In my testing I am finding that using SQLite's mmap support with
larger databases (final db size is ~10GB) to create the database (WAL
mode) is considerably (about three times) slower than no mmap.
The default max mmap limit is 2GB (sadly). Has anyon
Hi,
I have been testing sqlite mmap on FreeBSD/DragonFly and it works
perfectly well. (the package manager uses sqlite this is where I have
been testing it)
would it be possible to activate it?
https://people.freebsd.org/~bapt/freebsd-dragonfly-mmap.diff
Best regards,
Bapt
The behaviour depends on whether you map shared or not. If for map
shared multiple users can read and write to the file simultaneously. If
you have a situation where you access he same bytes you need to use some
form of synchronization, just as you do with read and write.
You can map for exc
John,
You seem pretty knowledgable regarding MMAP. I was wondering if you could help
me with this MMAP scenario:
I'm curious as to how the OS and multple processes interact regarding file
i/o and mmap.
Process A --- Writes to a file sequentially using either pwrite or kaio.
I
MMAP just lets you avoid one or two layers of buffering and APIs. If
you were to use fopen/fread you go to function calls then open/read plus
buffering and function calls then to to the VM to actually access the
data. Going direct to the VM and getting a pointer to the VM pages is
more effici
Hi John! Thanks for the reply!
I think that makes a good point that the vm page fault is probably
faster than the overhead of copying the data to a local buffer. So, page
fault or not, I think that's the way I'm going to do it.
Again, thanks very much for your input!
On 6/12/07, John Stanton <
Mitchell Vincent wrote:
Working with some data conversion here (that will eventually go into
an SQLite database). I'm hoping you IO wizards can offer some help on
a question that I've been trying to get answered.
I'm using Solaris 10 for this.
If I mmap a large file and use madvise with MADV_SE
Working with some data conversion here (that will eventually go into
an SQLite database). I'm hoping you IO wizards can offer some help on
a question that I've been trying to get answered.
I'm using Solaris 10 for this.
If I mmap a large file and use madvise with MADV_SEQUENTIAL and
MADV_WILLNEE
Christian Smith wrote:
At the risk of this turning into another argument between us...
On Thu, 20 Apr 2006, John Stanton wrote:
Q How do programs get to be slow and bloated?
A One small inefficiency at a time.
It is just as inefficient to send dynamically created content through a
series of
At the risk of this turning into another argument between us...
On Thu, 20 Apr 2006, John Stanton wrote:
>Q How do programs get to be slow and bloated?
>A One small inefficiency at a time.
>
>It is just as inefficient to send dynamically created content through a
>series of buffers as it is to s
Christian Smith wrote:
On Wed, 19 Apr 2006, John Stanton wrote:
Chris, Thanks for the very pertinent comments on segment linking.
I am not sending an Sqlite database. It is the output from my program,
principally assembled HTML/PDF/PostScript and similar.
I want to avoid buffer shadowing on
On Wed, 19 Apr 2006, John Stanton wrote:
>Chris, Thanks for the very pertinent comments on segment linking.
>
>I am not sending an Sqlite database. It is the output from my program,
>principally assembled HTML/PDF/PostScript and similar.
>
>I want to avoid buffer shadowing on the content my progr
Nuno Lucas wrote:
On 4/18/06, Christian Smith <[EMAIL PROTECTED]> wrote:
The OS has to track each MMAP segment, which is usually a linear linked
list sorted by virtual address. As most processes don't have that many
segments mapped (10's of segments would be considered a lot) the O(n)
linear se
Chris, Thanks for the very pertinent comments on segment linking.
I am not sending an Sqlite database. It is the output from my program,
principally assembled HTML/PDF/PostScript and similar.
I want to avoid buffer shadowing on the content my program creates. If
I use a malloc'd buffer it g
On 4/18/06, Christian Smith <[EMAIL PROTECTED]> wrote:
> The OS has to track each MMAP segment, which is usually a linear linked
> list sorted by virtual address. As most processes don't have that many
> segments mapped (10's of segments would be considered a lot) the O(n)
> linear search is not co
On Sun, 16 Apr 2006, John Stanton wrote:
>I wonder if members can help me with some advice. I have a program
>which is a multi-threaded application server with Sqlite embedded which
>runs on Unix and Windows. For an i/o buffer per thread I have the idea
>of using a mmap'd file so that it can be
On 4/16/06, John Stanton <[EMAIL PROTECTED]> wrote:
> John Stanton wrote:
> > I wonder if members can help me with some advice. I have a program
> > which is a multi-threaded application server with Sqlite embedded which
> > runs on Unix and Windows. For an i/o buffer per thread I have the ide
John Stanton
Cc: sqlite-users@sqlite.org
Betreff: Re[2]: [sqlite] MMAP
In my experience under windows, using memory mapped files is no faster than
opening and reading the file into memory then processing it. The limiting
factor is the disk drives. I use both methods and the choice of method is
alw
users@sqlite.org
Betreff: Re[2]: [sqlite] MMAP
In my experience under windows, using memory mapped files is no faster than
opening and reading the file into memory then processing it. The limiting
factor is the disk drives. I use both methods and the choice of method is
always a tossup. If the AP
In my experience under windows, using memory mapped files is no
faster than opening and reading the file into memory then processing
it. The limiting factor is the disk drives. I use both methods and the
choice of method is always a tossup. If the API I'm using will accept
an entire buffer for the
John Stanton wrote:
> I wonder if members can help me with some advice. I have a program
> which is a multi-threaded application server with Sqlite embedded which
> runs on Unix and Windows. For an i/o buffer per thread I have the idea
> of using a mmap'd file so that it can be transferred using
John Stanton <[EMAIL PROTECTED]> wrote:
> I wonder if members can help me with some advice. I have a program
> which is a multi-threaded application server with Sqlite embedded which
> runs on Unix and Windows. For an i/o buffer per thread I have the idea
> of using a mmap'd file so that it ca
First off: SQLite is not made to be scalable. If you are concerned
about performance, use something else.
The only performance issue you need to worry about with memory mapping
is running out of address space. BTW- I don't know about sendfile but
TransmitFile does not need memory mapping.
On 4/
I wonder if members can help me with some advice. I have a program
which is a multi-threaded application server with Sqlite embedded which
runs on Unix and Windows. For an i/o buffer per thread I have the idea
of using a mmap'd file so that it can be transferred using
sendfile/TransmitFile wi
33 matches
Mail list logo
