Re: [sqlite] the xRead method in sqlite3_io_methods
Here is the changed version a made for it: http://baiy.cn/tmp/sqlite.zip .
Search "by BaiYang - add piRealAmt" to go through all changes I have made.
With these simple changes, I could for example, write a simple VFS shim that
implement data encryption by only adding decrypt algorithm for xRead, adding
encrypt algorithm for xWrite, and forward ALL other methods to the underlayer
VFS.
On the other hand, without the piRealAmt argument, we need to re-implement a
complete VFS to add the encryption functionality.
--
Best Regards
BaiYang
baiy...@263.net.cn
http://baiy.cn
< END OF EMAIL > ****
发件人： BaiYang
发送时间： 2012-04-27 05:54
收件人： sqlite-users
主题： the xRead method in sqlite3_io_methods
We should add a new output argument for xRead method which is in
sqlite3_io_methods structure like this:
int (*xRead)(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst, int*
piRealAmt);
The new 'piRealAmt' could be used to report the actual read data, this argument
will greatly boost the ability of VFS Shims. For example, with this argument,
we can write a encryption shim easily.
It's very simple. Basically, you only need adding just one line code per xRead
driver like this:
}else{
pFile->lastErrno = 0; /* not a system error */
/* Unread parts of the buffer must be zero-filled */
memset(&((char*)pBuf)[got], 0, amt-got);
if (piRealAmt) *piRealAmt = (int)got; // <
return SQLITE_IOERR_SHORT_READ;
}
By using this simple extension, we can write code like this:
//! shim xRead driver for encryption
BAIY_LOCAL int
ReadMethod(
IN OUT sqlite3_file* piFile,
OUT void* pbBuf,
IN int nAmt,
IN sqlite_int64 nnOfst,
OPTIONAL OUT int* piRealAmt)
{
// =
// = Init, Guard
ENCVFS_FILE* piOriFile = reinterpret_cast(piFile);
if (piOriFile->btUnder.empty() || !piOriFile->thCipher.IsValid())
{
return SQLITE_IOERR_READ;
}
sqlite3_file* const piUnder = reinterpret_cast(
piOriFile->btUnder.ref()
);
// =
// = Read
int nReadLen;
piOriFile->btBuf.reserve(nAmt);
const int r = piUnder->pMethods->xRead(
piUnder, piOriFile->btBuf.ref(), nAmt, nnOfst, &nReadLen
);
switch (r)
{
case SQLITE_OK:
nReadLen = nAmt;
break;
case SQLITE_IOERR_SHORT_READ:
break;
default:
return r; // <
}
// =
// = Decrypt
try
{
piOriFile->thCipher->SeekDecryptor(nnOfst);
piOriFile->thCipher->IterDecrypt(
bySBn((BYTE*)pbBuf, nReadLen), piOriFile->btBuf
);
if (SQLITE_IOERR_SHORT_READ == r)
{
memset(((BYTE*)pbBuf)+nReadLen, 0, nAmt-nReadLen);
if (piRealAmt)
{
*piRealAmt = nReadLen;
}
return SQLITE_IOERR_SHORT_READ;
}
else
{
return SQLITE_OK;
}
}
catch (...)
{
LogError(byT("ReadMethod"));
return SQLITE_IOERR_READ;
}
}
--
Best Regards
BaiYang
baiy...@263.net.cn
http://baiy.cn
< END OF EMAIL >
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] the xRead method in sqlite3_io_methods
We should add a new output argument for xRead method which is in sqlite3_io_methods structure like this: int (*xRead)(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst, int* piRealAmt); The new 'piRealAmt' could be used to report the actual read data, this argument will greatly boost the ability of VFS Shims. For example, with this argument, we can write a encryption shim easily. -- Best Regards BaiYang baiy...@263.net.cn http://baiy.cn < END OF EMAIL > ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] the xRead method in sqlite3_io_methods
Ok, thanks for the reply :-)
--
Best Regards
BaiYang
baiy...@263.net.cn
http://baiy.cn
< END OF EMAIL >
From: Pavel Ivanov
Date: 2012-04-27 09:03
To: General Discussion of SQLite Database
Subject: Re: [sqlite] the xRead method in sqlite3_io_methods
> 1. The sqlite engine will discard ALL of the partially read data returned by
> xRead?
> For example, if the sqlite engine request 2096 bytes and xRead only got 1234
> bytes, does the engine will ALWAYS discard the 1234 bytes of data ?
> i.e: SQLITE_IOERR_SHORT_READ constantly means "no data" for sqlite engine ?
No, it doesn't mean "no data" for SQLite engine, but with
non-corrupted database it can happen only when there's no data. So if
SQLite requests 2096 bytes and xRead returns 1234 bytes SQLite will
detect database corruption either right away or a little bit later
(depending on when it will actually try to use the data that should
have been above 1234 bytes).
> 2. Are following warnings: "If xRead() returns SQLITE_IOERR_SHORT_READ it
> must also fill in the unread portions of the buffer with zeros. A VFS that
> fails to zero-fill short reads might seem to work. However, failure to
> zero-fill short reads will eventually lead to database corruption. ", Which
> listed in the official document: http://www.sqlite.org/c3ref/io_methods.html
> a joke ?
No, it's not a joke. It's a requirement that will help SQLite engine
to detect database corruption.
Pavel
On Thu, Apr 26, 2012 at 8:02 PM, BaiYang wrote:
>> They are not about header but they are essentially dealing with fatal
>> errors. First example is (as comment says) executed when database
>> tries to recover after a crashed process. And short read here means
>> that journal is corrupted and should be ignored. In the second example
>> if you follow the code further you'll see that SQLite assumes that
>> short read means 0 bytes was read and the whole page is filled with
>> zeros. That will mean that the page should be created.
>
> I don't know what you really want to explained, did you means:
>
> 1. The sqlite engine will discard ALL of the partially read data returned by
> xRead?
> For example, if the sqlite engine request 2096 bytes and xRead only got 1234
> bytes, does the engine will ALWAYS discard the 1234 bytes of data ?
> i.e: SQLITE_IOERR_SHORT_READ constantly means "no data" for sqlite engine ?
>
> 2. Are following warnings: "If xRead() returns SQLITE_IOERR_SHORT_READ it
> must also fill in the unread portions of the buffer with zeros. A VFS that
> fails to zero-fill short reads might seem to work. However, failure to
> zero-fill short reads will eventually lead to database corruption. ", Which
> listed in the official document: http://www.sqlite.org/c3ref/io_methods.html
> a joke ?
>
> I think the new argument is useful if any one is "no".
>
> --
> Best Regards
> BaiYang
> baiy...@263.net.cn
> http://baiy.cn
> < END OF EMAIL >
>
> From: Pavel Ivanov
> Date: 2012-04-27 07:34
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] the xRead method in sqlite3_io_methods
>> All these codes are dealing with the header?
>
> They are not about header but they are essentially dealing with fatal
> errors. First example is (as comment says) executed when database
> tries to recover after a crashed process. And short read here means
> that journal is corrupted and should be ignored. In the second example
> if you follow the code further you'll see that SQLite assumes that
> short read means 0 bytes was read and the whole page is filled with
> zeros. That will mean that the page should be created.
>
>
> Pavel
>
>
> On Thu, Apr 26, 2012 at 7:13 PM, BaiYang wrote:
>>> It is a fatal error. The only time it is okay is after the file is first
>>> created (zero length) ...
>> Really? But I have seen many location seems not like you sad. For example:
>>
>> In pager_playback:
>>}else if( rc==SQLITE_IOERR_SHORT_READ ){
>> /* If the journal has been truncated, simply stop reading and
>> ** processing the journal. This might happen if the journal was
>> ** not completely written and synced prior to a crash. In that
>> ** case, the database should have never been written in the
>> ** first place so it is OK to simply abandon the rollback. */
>> rc = SQLITE_OK;
>> goto end_playback;
>>
>> In readDbPage:
>> if( rc==SQLITE_OK && !isInWal ){
>>i64 iOffset = (pgno-1)*(i64)pPager->pageSize;
>>rc = sqlite3OsRead(pPager->fd, pPg->pData, p
Re: [sqlite] the xRead method in sqlite3_io_methods
> They are not about header but they are essentially dealing with fatal
> errors. First example is (as comment says) executed when database
> tries to recover after a crashed process. And short read here means
> that journal is corrupted and should be ignored. In the second example
> if you follow the code further you'll see that SQLite assumes that
> short read means 0 bytes was read and the whole page is filled with
> zeros. That will mean that the page should be created.
I don't know what you really want to explained, did you means:
1. The sqlite engine will discard ALL of the partially read data returned by
xRead?
For example, if the sqlite engine request 2096 bytes and xRead only got 1234
bytes, does the engine will ALWAYS discard the 1234 bytes of data ?
i.e: SQLITE_IOERR_SHORT_READ constantly means "no data" for sqlite engine ?
2. Are following warnings: "If xRead() returns SQLITE_IOERR_SHORT_READ it must
also fill in the unread portions of the buffer with zeros. A VFS that fails to
zero-fill short reads might seem to work. However, failure to zero-fill short
reads will eventually lead to database corruption. ", Which listed in the
official document: http://www.sqlite.org/c3ref/io_methods.html a joke ?
I think the new argument is useful if any one is "no".
--
Best Regards
BaiYang
baiy...@263.net.cn
http://baiy.cn
< END OF EMAIL >
From: Pavel Ivanov
Date: 2012-04-27 07:34
To: General Discussion of SQLite Database
Subject: Re: [sqlite] the xRead method in sqlite3_io_methods
> All these codes are dealing with the header?
They are not about header but they are essentially dealing with fatal
errors. First example is (as comment says) executed when database
tries to recover after a crashed process. And short read here means
that journal is corrupted and should be ignored. In the second example
if you follow the code further you'll see that SQLite assumes that
short read means 0 bytes was read and the whole page is filled with
zeros. That will mean that the page should be created.
Pavel
On Thu, Apr 26, 2012 at 7:13 PM, BaiYang wrote:
>> It is a fatal error. The only time it is okay is after the file is first
>> created (zero length) ...
> Really? But I have seen many location seems not like you sad. For example:
>
> In pager_playback:
>}else if( rc==SQLITE_IOERR_SHORT_READ ){
> /* If the journal has been truncated, simply stop reading and
> ** processing the journal. This might happen if the journal was
> ** not completely written and synced prior to a crash. In that
> ** case, the database should have never been written in the
> ** first place so it is OK to simply abandon the rollback. */
> rc = SQLITE_OK;
> goto end_playback;
>
> In readDbPage:
> if( rc==SQLITE_OK && !isInWal ){
>i64 iOffset = (pgno-1)*(i64)pPager->pageSize;
>rc = sqlite3OsRead(pPager->fd, pPg->pData, pgsz, iOffset);
>if( rc==SQLITE_IOERR_SHORT_READ ){
> rc = SQLITE_OK;
>}
> }
>
> And so on.
>
> All these codes are dealing with the header?
>
> --
> Best Regards
> BaiYang
> baiy...@263.net.cn
> http://baiy.cn
> < END OF EMAIL >
>
> From: Roger Binns
> Date: 2012-04-27 07:06
> To: General Discussion of SQLite Database
> Subject: Re: [sqlite] the xRead method in sqlite3_io_methods
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 26/04/12 15:27, BaiYang wrote:
>> I'm confused, does you mean a xRead should NEVER return a
>> "SQLITE_IOERR_SHORT_READ" ?
>
> It is a fatal error. The only time it is okay is after the file is first
> created (zero length) and SQLite tries to read the header page which
> doesn't exist at that point in time.
>
> At all other times a short error is fatal. The extended error code is so
> that the caller has more details than just I/O error.
>
> Roger
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
>
> iEYEARECAAYFAk+Z1O8ACgkQmOOfHg372QTyoACglrfe4j5Grsl2YzJ387bbquxE
> U4YAoOCoFPF425QNlEB3/tYntK+vTbHk
> =5wii
> -END PGP SIGNATURE-
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] the xRead method in sqlite3_io_methods
And according to the official document: If xRead() returns SQLITE_IOERR_SHORT_READ it must also fill in the unread portions of the buffer with zeros. A VFS that fails to zero-fill short reads might seem to work. However, failure to zero-fill short reads will eventually lead to database corruption. http://www.sqlite.org/c3ref/io_methods.html It's seems not as simple as you sad :-) -- Best Regards BaiYang baiy...@263.net.cn http://baiy.cn < END OF EMAIL > From: Roger Binns Date: 2012-04-27 07:06 To: General Discussion of SQLite Database Subject: Re: [sqlite] the xRead method in sqlite3_io_methods -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 26/04/12 15:27, BaiYang wrote: > I'm confused, does you mean a xRead should NEVER return a > "SQLITE_IOERR_SHORT_READ" ? It is a fatal error. The only time it is okay is after the file is first created (zero length) and SQLite tries to read the header page which doesn't exist at that point in time. At all other times a short error is fatal. The extended error code is so that the caller has more details than just I/O error. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk+Z1O8ACgkQmOOfHg372QTyoACglrfe4j5Grsl2YzJ387bbquxE U4YAoOCoFPF425QNlEB3/tYntK+vTbHk =5wii -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] the xRead method in sqlite3_io_methods
> It is a fatal error. The only time it is okay is after the file is first
> created (zero length) ...
Really? But I have seen many location seems not like you sad. For example:
In pager_playback:
}else if( rc==SQLITE_IOERR_SHORT_READ ){
/* If the journal has been truncated, simply stop reading and
** processing the journal. This might happen if the journal was
** not completely written and synced prior to a crash. In that
** case, the database should have never been written in the
** first place so it is OK to simply abandon the rollback. */
rc = SQLITE_OK;
goto end_playback;
In readDbPage:
if( rc==SQLITE_OK && !isInWal ){
i64 iOffset = (pgno-1)*(i64)pPager->pageSize;
rc = sqlite3OsRead(pPager->fd, pPg->pData, pgsz, iOffset);
if( rc==SQLITE_IOERR_SHORT_READ ){
rc = SQLITE_OK;
}
}
And so on.
All these codes are dealing with the header?
--
Best Regards
BaiYang
baiy...@263.net.cn
http://baiy.cn
< END OF EMAIL >
From: Roger Binns
Date: 2012-04-27 07:06
To: General Discussion of SQLite Database
Subject: Re: [sqlite] the xRead method in sqlite3_io_methods
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 26/04/12 15:27, BaiYang wrote:
> I'm confused, does you mean a xRead should NEVER return a
> "SQLITE_IOERR_SHORT_READ" ?
It is a fatal error. The only time it is okay is after the file is first
created (zero length) and SQLite tries to read the header page which
doesn't exist at that point in time.
At all other times a short error is fatal. The extended error code is so
that the caller has more details than just I/O error.
Roger
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAk+Z1O8ACgkQmOOfHg372QTyoACglrfe4j5Grsl2YzJ387bbquxE
U4YAoOCoFPF425QNlEB3/tYntK+vTbHk
=5wii
-END PGP SIGNATURE-
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] the xRead method in sqlite3_io_methods
> If your internal code results in less data being read then you'll need to > keep repeating until you have the amount requested. I'm confused, does you mean a xRead should NEVER return a "SQLITE_IOERR_SHORT_READ" ? -- Best Regards BaiYang baiy...@263.net.cn http://baiy.cn < END OF EMAIL > From: Roger Binns Date: 2012-04-27 06:11 To: General Discussion of SQLite Database Subject: Re: [sqlite] the xRead method in sqlite3_io_methods -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 26/04/12 14:54, BaiYang wrote: > The new 'piRealAmt' could be used to report the actual read data, Why? The requirements are that you read the amount of data requested or return an error. If your internal code results in less data being read then you'll need to keep repeating until you have the amount requested. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk+ZyCYACgkQmOOfHg372QRDIQCgyo8nARHPfB3KUYjv/vpS5xWz ZTkAni/JaeSFlNVJjO0xRHTqqtxCyVib =cSZL -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] the xRead method in sqlite3_io_methods
We should add a new output argument for xRead method which is in
sqlite3_io_methods structure like this:
int (*xRead)(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst, int*
piRealAmt);
The new 'piRealAmt' could be used to report the actual read data, this argument
will greatly boost the ability of VFS Shims. For example, with this argument,
we can write a encryption shim easily.
It's very simple. Basically, you only need adding just one line code per xRead
driver like this:
}else{
pFile->lastErrno = 0; /* not a system error */
/* Unread parts of the buffer must be zero-filled */
memset(&((char*)pBuf)[got], 0, amt-got);
if (piRealAmt) *piRealAmt = (int)got; // <
return SQLITE_IOERR_SHORT_READ;
}
By using this simple extension, we can write code like this:
//! shim xRead driver for encryption
BAIY_LOCAL int
ReadMethod(
IN OUT sqlite3_file* piFile,
OUT void* pbBuf,
IN int nAmt,
IN sqlite_int64 nnOfst,
OPTIONAL OUT int* piRealAmt)
{
// =
// = Init, Guard
ENCVFS_FILE* piOriFile = reinterpret_cast(piFile);
if (piOriFile->btUnder.empty() || !piOriFile->thCipher.IsValid())
{
return SQLITE_IOERR_READ;
}
sqlite3_file* const piUnder = reinterpret_cast(
piOriFile->btUnder.ref()
);
// =
// = Read
int nReadLen;
piOriFile->btBuf.reserve(nAmt);
const int r = piUnder->pMethods->xRead(
piUnder, piOriFile->btBuf.ref(), nAmt, nnOfst, &nReadLen
);
switch (r)
{
case SQLITE_OK:
nReadLen = nAmt;
break;
case SQLITE_IOERR_SHORT_READ:
break;
default:
return r; // <
}
// =
// = Decrypt
try
{
piOriFile->thCipher->SeekDecryptor(nnOfst);
piOriFile->thCipher->IterDecrypt(
bySBn((BYTE*)pbBuf, nReadLen), piOriFile->btBuf
);
if (SQLITE_IOERR_SHORT_READ == r)
{
memset(((BYTE*)pbBuf)+nReadLen, 0, nAmt-nReadLen);
if (piRealAmt)
{
*piRealAmt = nReadLen;
}
return SQLITE_IOERR_SHORT_READ;
}
else
{
return SQLITE_OK;
}
}
catch (...)
{
LogError(byT("ReadMethod"));
return SQLITE_IOERR_READ;
}
}
--
Best Regards
BaiYang
baiy...@263.net.cn
http://baiy.cn
< END OF EMAIL >
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
