Re: [sqlite] SQLite working with C++'s iostream
On Mon, Oct 4, 2010 at 1:53 AM, Roger Binns wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 10/03/2010 10:01 PM, Max Vlasov wrote: > > On Sun, Oct 3, 2010 at 7:21 PM, Roger Binns > wrote: > > > >> > >> Also note that xTruncate may be called to make a file longer. > >> > >> > > Roger, are you sure about that? > > That has certainly been stated in the past. Unfortunately the current > io_methods documentation doesn't say anything about xTruncate. > > Looking through the source, the vast majority of cases would indeed result > in a shorter file with writes used to extend. But there may be lengthening > from walCheckpoint() and possibly as part of the backup process. > > In any event unless there is an explicit guarantee from the SQLite authors > that xTruncate will never be called to lengthen a file (including in future > versions) then it should be something that should be checked for and > errored > or implemented as appropriate. > I'm not willing to say we will support this forever, but recent versions of SQLite do work correctly if xTruncate is a no-op. I think the support for this began with version 3.7.0. If xTruncate is a no-op, then the VACUUM command will not actually reduce the size of the database file, of course. But everything else should work as expected. The xTruncate=noop support was added for a popular consumer gadget that uses SQLite on an operating system that does not support file truncation. > > Roger > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkypa/EACgkQmOOfHg372QRugwCguKqqfXhf2+/jrwDYnYwid7CY > mjkAn1JnTYIhlKey7YW6QumIWe4oLwJR > =5hwL > -END PGP SIGNATURE- > ___ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite working with C++'s iostream
On Sun, Oct 03, 2010 at 06:31:41PM +0200, Pierre Krieger scratched on the wall: > But the main reason why I would use streams is for other things like > reading data from a socket or decrypting a file on-the-fly for example > (these are just ideas) SQLite has some very specific requirements for VFS modules, especially in regards to locking. It is likely that anything that is developed would need to be custom written with those factors in mind. Also be aware that VFS modules can be "stacked". For example, you could write an encryption/decryption VFS that modified the memory buffers, but otherwise passed through all the actual I/O calls, as well as the locking calls, to the existing UNIX or Windows VFS module. In that way, much of the flexibility of a stream based systems is already there, even if it requires a somewhat different implementation. -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite working with C++'s iostream
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/03/2010 10:01 PM, Max Vlasov wrote: > On Sun, Oct 3, 2010 at 7:21 PM, Roger Binns wrote: > >> >> Also note that xTruncate may be called to make a file longer. >> >> > Roger, are you sure about that? That has certainly been stated in the past. Unfortunately the current io_methods documentation doesn't say anything about xTruncate. Looking through the source, the vast majority of cases would indeed result in a shorter file with writes used to extend. But there may be lengthening from walCheckpoint() and possibly as part of the backup process. In any event unless there is an explicit guarantee from the SQLite authors that xTruncate will never be called to lengthen a file (including in future versions) then it should be something that should be checked for and errored or implemented as appropriate. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkypa/EACgkQmOOfHg372QRugwCguKqqfXhf2+/jrwDYnYwid7CY mjkAn1JnTYIhlKey7YW6QumIWe4oLwJR =5hwL -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite working with C++'s iostream
On Sun, Oct 3, 2010 at 7:21 PM, Roger Binns wrote: > > Also note that xTruncate may be called to make a file longer. > > Roger, are you sure about that? My own experience with VFS showed that expanding was always handled by xWrite pointing to the offset outside the current container size and after the extensive usage of a VFS-filtered query, the first sql query that actually touched xTruncate was VACUUM. Max Vlasov maxerist.net ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite working with C++'s iostream
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/03/2010 10:03 AM, Pierre Krieger wrote: > For example a stream which handles a simple ROT-13 "encryption" can > be written in ten minutes (if you know the internals of the iostream library) My Python wrapper has this functionality as an example although I use XOR rather than ROT-13. It is a trivial number of lines of code: http://apidoc.apsw.googlecode.com/hg/example.html#example-vfs But the reason why this works well is that I provide a way of "inheriting" from an existing VFS implementation. So all you have to do is override xOpen to return an xor'd file instance and then override the latter's xRead and xWrite to xor the data before passing to the parents xRead/xWrite. With VFS it is good practise to punt to as much existing code as possible, only providing overridden methods where your VFS adds value. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyoxQYACgkQmOOfHg372QS6pwCcCmBAlb9CJw24dMO1Lg5woyDI oYcAoNAvr5qXybYxufe/eSZ/z25AmQor =Cy93 -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite working with C++'s iostream
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/03/2010 09:31 AM, Pierre Krieger wrote: > Anyway I don't think about using this code in a real program for the > moment, I just made it because of my love for modularity and because it > looked like a nice idea It is a good idea and the use cases you gave (networked data, compression etc) are good matches for using VFS. The problem side is the io::stream API is not a particularly good match for what SQLite needs from a VFS. You'd likely be ok if the data only ever needs to be read, but if that is the case then presenting the data as a web service would be better in the long term. For writable data you do have to implement locking, changing file size, disk sync etc which io:stream provides no help for. (Incidentally your implementation of xSync bears no resemblance with actually getting the data onto disk platters :-) You should also ideally make sure that if the data source is a local file then the same file being opened by regular SQLite will be ok and not cause corruption. Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyowuUACgkQmOOfHg372QQQZACcCPsW9EqoJXXoXPrJqB5iCgkb N0gAoJHcVAdVkLgfDuTTcIxO1nv3mlR5 =zq2l -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite working with C++'s iostream
> That's going to be problematic, because almost all of SQLite's usage > of underlying I/O is block-style, not stream-style, more akin to > accessing elements in a persistent byte array (with other practical > matters added on, of course) than to reading and writing streams. > You'd probably be better off just writing temporary files in both of > the cases you mention, unless (in the latter case) you actually have > per-block (or similar) crypto going on rather than purely streaming. > > (You do use seekp, but some underlying streams might not support it.) I know this, in fact that's why I wrote that they are just ideas (reading from a socket was a bad idea) But a derivate of iostream could for example do lazy-decrypting by decrypting only blocks of data that are required, and would store all the writes in a buffer that is flushed (ie. re-encrypted) when the sync function is called For example a stream which handles a simple ROT-13 "encryption" can be written in ten minutes (if you know the internals of the iostream library) I really think there are a lot of things that you can do with streams ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite working with C++'s iostream
Quoth Pierre Krieger , on 2010-10-03 18:31:41 +0200: > But the main reason why I would use streams is for other things like > reading data from a socket or decrypting a file on-the-fly for example > (these are just ideas) That's going to be problematic, because almost all of SQLite's usage of underlying I/O is block-style, not stream-style, more akin to accessing elements in a persistent byte array (with other practical matters added on, of course) than to reading and writing streams. You'd probably be better off just writing temporary files in both of the cases you mention, unless (in the latter case) you actually have per-block (or similar) crypto going on rather than purely streaming. (You do use seekp, but some underlying streams might not support it.) ---> Drake Wilson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite working with C++'s iostream
Thanks for answering > You claim that documentation is missing. Specifically what was not > documented? Well I was referring to the online documentation (sqlite.org/c3ref/vfs.html and sqlite.org/c3ref/io_methods.html) which don't explain the effects, possible return codes, etc. of some of the function. I recognize this was just an excuse for implementing them later (as your source code is impeccable) > The first major reason why there would be a problem is that you do not > support locking so if the stream ever referred to a file it would be > possible to corrupt it should another process access the file using the > SQLite library. I think that this is the stream creator's problem! Of course it would not be very efficient because the file would have to be locked for writing all the time, even if you just read it But the main reason why I would use streams is for other things like reading data from a socket or decrypting a file on-the-fly for example (these are just ideas) > Secondly you need to prevent exceptions from happening in any of your VFS > callbacks as they'll unwind the stack and not let SQLite's error recovery run. I don't think it's a problem either, a try-catch block around each function's body will do the trick Anyway I don't think about using this code in a real program for the moment, I just made it because of my love for modularity and because it looked like a nice idea ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] SQLite working with C++'s iostream
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/03/2010 12:25 AM, Pierre KRIEGER wrote: > I made my own implementation You claim that documentation is missing. Specifically what was not documented? BTW for SQLITE_GET_LOCKPROXYFILE just Google "sqlite SQLITE_GET_LOCKPROXYFILE". You should just return SQLITE_ERROR for any filecontrol you do not handle. > * when you try to open a file named 'f' for example, what other files are > being > accessed? I noticed 'f-journal' and 'f-wal', are there others? f-shm. But wal and shm are only for WAL mode which requires VFS version 2 and you only implement version 1. Note that it is possible for another connection to have opened the database in WAL mode and then your code being called with the same file so SQLite may sniff around the WAL files but not use them. > * it is not possible to truncate a stream, so is there a way to be sure that > SQLite won't call the xTruncate function? Not really. In general you should be able to get by if you write 1024 bytes of zero bytes from the truncation point onwards. Also note that xTruncate may be called to make a file longer. > * when you close a file other than the main database, it gets virtually > deleted > (even if it didn't have the temporary flag) does SQLite always accept > this? Zero out the beginning 1024 bytes of the deleted file. > * do you think my code has a chance to be reliable in a real-world project? Not even close. The way you would test this is by using the SQLite test suite telling it to use your vfs. The first major reason why there would be a problem is that you do not support locking so if the stream ever referred to a file it would be possible to corrupt it should another process access the file using the SQLite library. Secondly you need to prevent exceptions from happening in any of your VFS callbacks as they'll unwind the stack and not let SQLite's error recovery run. For my Python wrapper I provide an API for VFS and a second one for VFSFile which can then be implemented as needed by the developer. I'd recommend a similar approach rather than trying to glue to io::stream directly. My documentation may also provide some insight: http://apidoc.apsw.googlecode.com/hg/vfs.html Roger -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyon2IACgkQmOOfHg372QRIHwCgteuP7uL0oXOtk3aQ3HnkI+JK DHIAoLy/m55y0AX7q0Fkbt73fp5F3FL+ =aWfE -END PGP SIGNATURE- ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
