[sqlite] Converting in-memory sqlite database to char array
Hi all, I have an in-memory sqlite database which I want to convert to a simple char array, to send over a communication protocol. I want to do this preferably without using any serialization library. Basically I want to do what backup api calls does, but instead of copying database to another database, I will be copying it to a char array/string/stream (whatever you may call it). ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Converting in-memory sqlite database to char array
As a first (simple) approach, I might use the standard backup API to back up to a temp file, then stream that file byte by byte over the communication protocol. I'm sure there may be other more direct-to-memory approaches, perhaps using a custom VFS. However, this approach should be simple and easy, and would not require any special serialization library-- just standard file I/O. On Thu, Nov 29, 2012 at 8:19 AM, Map Scape halukcy...@gmail.com wrote: Hi all, I have an in-memory sqlite database which I want to convert to a simple char array, to send over a communication protocol. I want to do this preferably without using any serialization library. Basically I want to do what backup api calls does, but instead of copying database to another database, I will be copying it to a char array/string/stream (whatever you may call it). ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Converting in-memory sqlite database to char array
And if you want to improve latency you can use fifo's on Unix or anonymous pipes on Windows and run a thread to send your data while it's writing since those methods are synchronous. man popen (you open write in one thread and open a read in another) http://msdn.microsoft.com/en-us/library/windows/desktop/aa365141%28v=vs.85%29.aspx On Windows you get 2 handles that you pass the read handle to your other thread. Remember to send a 2nd item (last packet) with how many bytes you sent so the client knows it got what it was supposed to. Otherwise you're sure to get a truncated db some time and die on the client. Michael D. Black Senior Scientist Advanced Analytics Directorate Advanced GEOINT Solutions Operating Unit Northrop Grumman Information Systems From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Eric Minbiole [eminbi...@gmail.com] Sent: Thursday, November 29, 2012 7:53 AM To: General Discussion of SQLite Database Subject: EXT :Re: [sqlite] Converting in-memory sqlite database to char array As a first (simple) approach, I might use the standard backup API to back up to a temp file, then stream that file byte by byte over the communication protocol. I'm sure there may be other more direct-to-memory approaches, perhaps using a custom VFS. However, this approach should be simple and easy, and would not require any special serialization library-- just standard file I/O. On Thu, Nov 29, 2012 at 8:19 AM, Map Scape halukcy...@gmail.com wrote: Hi all, I have an in-memory sqlite database which I want to convert to a simple char array, to send over a communication protocol. I want to do this preferably without using any serialization library. Basically I want to do what backup api calls does, but instead of copying database to another database, I will be copying it to a char array/string/stream (whatever you may call it). ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Converting in-memory sqlite database to char array
writing to a file is indeed the easy way, but I would rather not involve the filesystem if possible. yet, I guess that will be the way I will choose at the end. I also had a look at VFS, but an easier method would have been better of course. On Thu, Nov 29, 2012 at 2:53 PM, Eric Minbiole eminbi...@gmail.com wrote: As a first (simple) approach, I might use the standard backup API to back up to a temp file, then stream that file byte by byte over the communication protocol. I'm sure there may be other more direct-to-memory approaches, perhaps using a custom VFS. However, this approach should be simple and easy, and would not require any special serialization library-- just standard file I/O. On Thu, Nov 29, 2012 at 8:19 AM, Map Scape halukcy...@gmail.com wrote: Hi all, I have an in-memory sqlite database which I want to convert to a simple char array, to send over a communication protocol. I want to do this preferably without using any serialization library. Basically I want to do what backup api calls does, but instead of copying database to another database, I will be copying it to a char array/string/stream (whatever you may call it). ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Converting in-memory sqlite database to char array
On Thu, Nov 29, 2012 at 02:05:02PM +, Black, Michael (IS) scratched on the wall: And if you want to improve latency you can use fifo's on Unix or anonymous pipes on Windows and run a thread to send your data while it's writing since those methods are synchronous. I would not assume the backup API writes the file front to back, especially if the database is modified while the backup is taking place. A custom VFS that just writes the file to a big chunk of memory makes the most sense. -j -- Jay A. Kreibich J A Y @ K R E I B I.C H Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable. -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Converting in-memory sqlite database to char array
I thought a backup was using a snapshot and locking the database? Hadn't considered random access though which I'd wager it does do on write. Michael D. Black Senior Scientist Advanced Analytics Directorate Advanced GEOINT Solutions Operating Unit Northrop Grumman Information Systems From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] on behalf of Jay A. Kreibich [j...@kreibi.ch] Sent: Thursday, November 29, 2012 8:37 AM To: General Discussion of SQLite Database Subject: EXT :Re: [sqlite] Converting in-memory sqlite database to char array On Thu, Nov 29, 2012 at 02:05:02PM +, Black, Michael (IS) scratched on the wall: And if you want to improve latency you can use fifo's on Unix or anonymous pipes on Windows and run a thread to send your data while it's writing since those methods are synchronous. I would not assume the backup API writes the file front to back, especially if the database is modified while the backup is taking place. A custom VFS that just writes the file to a big chunk of memory makes the most sense. -j -- Jay A. Kreibich J A Y @ K R E I B I.C H Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable. -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Converting in-memory sqlite database to char array
On Thu, Nov 29, 2012 at 02:39:49PM +, Black, Michael (IS) scratched on the wall: I thought a backup was using a snapshot and locking the database? No... the source DB remains available. That's largely the point of the API. In fact, the full name is the Online Backup API. The fact that it can also be used to copy in-memory DBs is more of a side benefit, even if it was a much needed benefit. http://www.sqlite.org/c3ref/backup_finish.html SQLite holds a write transaction open on the destination database file for the duration of the backup operation. The source database is read-locked only while it is being read; it is not locked continuously for the entire backup operation. Thus, the backup may be performed on a live source database without preventing other database connections from reading or writing to the source database while the backup is underway. If the source database is modified by the same connection doing the backup, the page updates are written to both DBs. If the source DB is modified by any other connection, the backup automatically restarts. Because it is easy to imagine a case when the backup gets caught in a restart loop, some people choose to make the backup a more atomic operation by having the backup step function copy all the pages in one go. In that case it is likely that the majority of pages are written out in-order, but I wouldn't want to bank on that. -j -- Jay A. Kreibich J A Y @ K R E I B I.C H Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable. -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
