Re: [sqlite] demovfs question

[Neo Anderson]

Thanks.


> From: d...@sqlite.org
> Date: Wed, 20 Mar 2013 12:48:45 -0400
> To: sqlite-users@sqlite.org
> Subject: Re: [sqlite] demovfs question
>
> On Wed, Mar 20, 2013 at 12:45 PM, Neo Anderson <neo_in_mat...@msn.com>wrote:
>
> > Thanks for your detailed explanation. I think I've got my answers for my 2
> > questions.
> >
> > But I've got another question:
> >
> > What happens if there is a power failure when the VFS is in middle of
> > writing journal data to disk but has not yet finished?
> >
> > Let's say the VFS has finished writing a half of integer field and now
> > power is down. Can sqlite handle this?
> >
>
> Yes. See http://www.sqlite.org/testing.html#crashtesting and
> http://www.sqlite.org/atomiccommit.html for further insight into how this
> works.
>
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users  
>   
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] demovfs question

[Richard Hipp]

On Wed, Mar 20, 2013 at 12:45 PM, Neo Anderson wrote:

> Thanks for your detailed explanation. I think I've got my answers for my 2
> questions.
>
> But I've got another question:
>
> What happens if there is a power failure when the VFS is in middle of
> writing journal data to disk but has not yet finished?
>
> Let's say the VFS has finished writing a half of integer field and now
> power is down. Can sqlite handle this?
>

Yes.  See http://www.sqlite.org/testing.html#crashtesting and
http://www.sqlite.org/atomiccommit.html for further insight into how this
works.


-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] demovfs question

[Neo Anderson]

Thanks for your detailed explanation. I think I've got my answers for my 2 
questions.

But I've got another question:

What happens if there is a power failure when the VFS is in middle of writing 
journal data to disk but has not yet finished?

Let's say the VFS has finished writing a half of integer field and now power is 
down. Can sqlite handle this?


> From: slav...@bigfraud.org
> Date: Wed, 20 Mar 2013 14:02:53 +
> To: sqlite-users@sqlite.org
> Subject: Re: [sqlite] demovfs question
>
>
> On 20 Mar 2013, at 10:17am, Neo Anderson <neo_in_mat...@msn.com> wrote:
>
> > Sorry, what do you mean on question 2?
> >
> > Can I use the buffered fopen family functions or not?
>
> I believe Dan is concerned about some issues with ACID and crash-recovery 
> which will be introduced if you add an extra level of buffering.
>
> To ensure data integrity is is vital that the bits on the disk are a perfect 
> reflection of the data in the database. This ensures that two processes 
> trying to access the same data agree on what the data is, and also that if a 
> process crashes while it's writing data to the database, the files on the 
> disk (which are what's going to be found when the database is next opened) 
> correctly reflect the history of committed transactions and are as up-to-date 
> as possible.
>
> By buffering file operations you are introducing a set of delays to your 
> operations. Consequently some of the 'state' of your data is neither 
> committed nor uncommitted, but stuck somewhere in buffers. A DBMS which takes 
> great pains to be as much ACID
>
> <http://en.wikipedia.org/wiki/ACID>
>
> as possible is one place where you would want to avoid buffering.
>
> So yes, technically, SQLite won't throw a fit if you use buffered calls. But 
> using them will break ACID in cases of simultaneous access and recovery from 
> crashes, power loss, broken cables, etc.. So it shouldn't be used in anything 
> except conditions where the programmers knows everything about where their 
> program will be used -- perhaps a one-off program which will only ever be run 
> on the programmers own personal hardware.
>
> If you have written unbuffered code and it is /too/ slow to be useful, by all 
> means get back to us and we can suggest things to tackle. But if rather than 
> /too/ slow, it is just a bit slower than you think it could be, then 
> buffering is not one of the best things to look at.
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users  
>   
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] demovfs question

[Simon Slavin]

On 20 Mar 2013, at 10:17am, Neo Anderson  wrote:

> Sorry, what do you mean on question 2?
> 
> Can I use the buffered fopen family functions or not?

I believe Dan is concerned about some issues with ACID and crash-recovery which 
will be introduced if you add an extra level of buffering.

To ensure data integrity is is vital that the bits on the disk are a perfect 
reflection of the data in the database.  This ensures that two processes trying 
to access the same data agree on what the data is, and also that if a process 
crashes while it's writing data to the database, the files on the disk (which 
are what's going to be found when the database is next opened) correctly 
reflect the history of committed transactions and are as up-to-date as possible.

By buffering file operations you are introducing a set of delays to your 
operations.  Consequently some of the 'state' of your data is neither committed 
nor uncommitted, but stuck somewhere in buffers.  A DBMS which takes great 
pains to be as much ACID



as possible is one place where you would want to avoid buffering.

So yes, technically, SQLite won't throw a fit if you use buffered calls.  But 
using them will break ACID in cases of simultaneous access and recovery from 
crashes, power loss, broken cables, etc..  So it shouldn't be used in anything 
except conditions where the programmers knows everything about where their 
program will be used -- perhaps a one-off program which will only ever be run 
on the programmers own personal hardware.

If you have written unbuffered code and it is /too/ slow to be useful, by all 
means get back to us and we can suggest things to tackle.  But if rather than 
/too/ slow, it is just a bit slower than you think it could be, then buffering 
is not one of the best things to look at.

Simon.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] demovfs question

[Dan Kennedy]

On 03/20/2013 05:17 PM, Neo Anderson wrote:

Sorry, what do you mean on question 2?

Can I use the buffered fopen family functions or not?


You probably can. I don't think you should though.






Date: Wed, 20 Mar 2013 17:13:26 +0700
From: danielk1...@gmail.com
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] demovfs question

On 03/20/2013 05:00 PM, Neo Anderson wrote:

I'm studying the demovfs listed on http://www.sqlite.org/vfs.html.

I've got a few questions:

1. By reading the comments, it seems that file read/write operations
must be buffered if it is a journal file. Is this required by sqlite
design?


No. Not required.


2. The demo uses nonbuffered open/read/write function to talk with
OS. Can I use the buffered fopen/fread/fwrite functions and thus I
don't bother with buffering management?


The user-space buffering seems quite likely to introduce confusing
bugs. Probably easiest not to.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users


___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users



___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] demovfs question

[Neo Anderson]

Sorry, what do you mean on question 2?

Can I use the buffered fopen family functions or not?


> Date: Wed, 20 Mar 2013 17:13:26 +0700
> From: danielk1...@gmail.com
> To: sqlite-users@sqlite.org
> Subject: Re: [sqlite] demovfs question
>
> On 03/20/2013 05:00 PM, Neo Anderson wrote:
> > I'm studying the demovfs listed on http://www.sqlite.org/vfs.html.
> >
> > I've got a few questions:
> >
> > 1. By reading the comments, it seems that file read/write operations
> > must be buffered if it is a journal file. Is this required by sqlite
> > design?
>
> No. Not required.
>
> > 2. The demo uses nonbuffered open/read/write function to talk with
> > OS. Can I use the buffered fopen/fread/fwrite functions and thus I
> > don't bother with buffering management?
>
> The user-space buffering seems quite likely to introduce confusing
> bugs. Probably easiest not to.
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users  
>   
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] demovfs question

[Dan Kennedy]

On 03/20/2013 05:00 PM, Neo Anderson wrote:

I'm studying the demovfs listed on http://www.sqlite.org/vfs.html.

I've got a few questions:

1. By reading the comments, it seems that file read/write operations
must be buffered if it is a journal file. Is this required by sqlite
design?


No. Not required.


2. The demo uses nonbuffered open/read/write function to talk with
OS. Can I use the buffered fopen/fread/fwrite functions and thus I
don't bother with buffering management?


The user-space buffering seems quite likely to introduce confusing
bugs. Probably easiest not to.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] demovfs question

[Neo Anderson]

I'm studying the demovfs listed on http://www.sqlite.org/vfs.html.

I've got a few questions:

1. By reading the comments, it seems that file read/write operations must be 
buffered if it is a journal file. Is this required by sqlite design?

2. The demo uses nonbuffered open/read/write function to talk with OS. Can I 
use the buffered fopen/fread/fwrite functions and thus I don't bother with 
buffering management? 
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users