Re: [sqlite] LSM1 extension
On 8/9/17, Jens Alfke wrote: > > One thing I’m unclear on: Will the LSM1 extension in SQLite3 be able to plug > in at a low level (replacing the b-tree engine), so one can use it with all > the existing features like relational tables and SQL queries; or will it > provide only a basic key/value store API? It is just a separate key/value store using a different file. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
> On Aug 9, 2017, at 8:23 AM, Nico Williams wrote: > > I must say though, SQLite4 is *brilliant*. I do hope that it either > gets completed and replaces SQLite3, or has all its good ideas folded > into SQLite3. The LSM1 work in SQLite3 seems to point towords the > latter. One thing I’m unclear on: Will the LSM1 extension in SQLite3 be able to plug in at a low level (replacing the b-tree engine), so one can use it with all the existing features like relational tables and SQL queries; or will it provide only a basic key/value store API? (Sorry if it’s off-topic to discuss future features here. I know there’s a development-focused list but I think I’d be way beyond my depth there. LSM1 is potentially very, very relevant to the project I’m working on, but that depends on how it ends up integrated, which is why I ask.) —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
On Wed, Aug 09, 2017 at 08:26:51AM -0500, Charles Leifer wrote: > sqlite4's future is uncertain. It's not, as I understand, meant to replace > sqlite3 any time soon. I think it was more of a place to try out new ideas > for implementations. I must say though, SQLite4 is *brilliant*. I do hope that it either gets completed and replaces SQLite3, or has all its good ideas folded into SQLite3. The LSM1 work in SQLite3 seems to point towords the latter. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
sqlite4's future is uncertain. It's not, as I understand, meant to replace sqlite3 any time soon. I think it was more of a place to try out new ideas for implementations. On Wed, Aug 9, 2017 at 3:10 AM, x wrote: > Thanks Charles. Is sqlite4 available yet? > > From: Charles Leifer<mailto:colei...@gmail.com> > Sent: 08 August 2017 21:21 > To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org> > Subject: Re: [sqlite] LSM1 extension > > There's some information that may be of interest on the sqlite4 wiki: > > * Design overview: https://sqlite.org/src4/doc/trunk/www/lsm.wiki > * User guide: https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki > * API: https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki > > My interest in this particular feature stems from my having written a set > of python bindings to the sqlite4 implementation (to be used as a sort of > embedded ordered key/value storage engine). I've wondered about the > possibility of exposing these APIs as a virtual table, and so seeing this > extension really excited me. So obviously my interest is much greater than > that of the average SQLite user :) > > On Tue, Aug 8, 2017 at 12:54 PM, x wrote: > > > Thanks Richard. > > > > From: Richard Hipp<mailto:d...@sqlite.org> > > Sent: 08 August 2017 18:47 > > To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org> > > Subject: Re: [sqlite] LSM1 extension > > > > On 8/8/17, x wrote: > > > I keep reading about this LSM1 but can’t find anything about it beyond > > > tickets. Any links? > > > > Baseline technology: https://en.wikipedia.org/wiki/ > > Log-structured_merge-tree > > > > No documentation yet on the LSM1 implementation in SQLite. > > > > -- > > D. Richard Hipp > > d...@sqlite.org > > ___ > > sqlite-users mailing list > > sqlite-users@mailinglists.sqlite.org > > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > > > > ___ > > sqlite-users mailing list > > sqlite-users@mailinglists.sqlite.org > > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
Thanks Charles. Is sqlite4 available yet? From: Charles Leifer<mailto:colei...@gmail.com> Sent: 08 August 2017 21:21 To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org> Subject: Re: [sqlite] LSM1 extension There's some information that may be of interest on the sqlite4 wiki: * Design overview: https://sqlite.org/src4/doc/trunk/www/lsm.wiki * User guide: https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki * API: https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki My interest in this particular feature stems from my having written a set of python bindings to the sqlite4 implementation (to be used as a sort of embedded ordered key/value storage engine). I've wondered about the possibility of exposing these APIs as a virtual table, and so seeing this extension really excited me. So obviously my interest is much greater than that of the average SQLite user :) On Tue, Aug 8, 2017 at 12:54 PM, x wrote: > Thanks Richard. > > From: Richard Hipp<mailto:d...@sqlite.org> > Sent: 08 August 2017 18:47 > To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org> > Subject: Re: [sqlite] LSM1 extension > > On 8/8/17, x wrote: > > I keep reading about this LSM1 but can’t find anything about it beyond > > tickets. Any links? > > Baseline technology: https://en.wikipedia.org/wiki/ > Log-structured_merge-tree > > No documentation yet on the LSM1 implementation in SQLite. > > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
There's some information that may be of interest on the sqlite4 wiki: * Design overview: https://sqlite.org/src4/doc/trunk/www/lsm.wiki * User guide: https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki * API: https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki My interest in this particular feature stems from my having written a set of python bindings to the sqlite4 implementation (to be used as a sort of embedded ordered key/value storage engine). I've wondered about the possibility of exposing these APIs as a virtual table, and so seeing this extension really excited me. So obviously my interest is much greater than that of the average SQLite user :) On Tue, Aug 8, 2017 at 12:54 PM, x wrote: > Thanks Richard. > > From: Richard Hipp<mailto:d...@sqlite.org> > Sent: 08 August 2017 18:47 > To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org> > Subject: Re: [sqlite] LSM1 extension > > On 8/8/17, x wrote: > > I keep reading about this LSM1 but can’t find anything about it beyond > > tickets. Any links? > > Baseline technology: https://en.wikipedia.org/wiki/ > Log-structured_merge-tree > > No documentation yet on the LSM1 implementation in SQLite. > > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
Thanks Richard. From: Richard Hipp<mailto:d...@sqlite.org> Sent: 08 August 2017 18:47 To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org> Subject: Re: [sqlite] LSM1 extension On 8/8/17, x wrote: > I keep reading about this LSM1 but can’t find anything about it beyond > tickets. Any links? Baseline technology: https://en.wikipedia.org/wiki/Log-structured_merge-tree No documentation yet on the LSM1 implementation in SQLite. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
On 8/8/17, x wrote: > I keep reading about this LSM1 but can’t find anything about it beyond > tickets. Any links? Baseline technology: https://en.wikipedia.org/wiki/Log-structured_merge-tree No documentation yet on the LSM1 implementation in SQLite. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
I keep reading about this LSM1 but can’t find anything about it beyond tickets. Any links? From: Jens Alfke<mailto:j...@mooseyard.com> Sent: 08 August 2017 18:04 To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org> Subject: Re: [sqlite] LSM1 extension > On Aug 4, 2017, at 9:12 PM, Charles Leifer wrote: > > Right -- my question is still unanswered as to why the code was merged in. It’s not uncommon in software engineering to put unfinished long-running feature work on the main development branch but disable it with a flag. (It was standard policy in one of the projects I worked on at Google.) This helps keep the mainline code and unfinished code from diverging, and it simplifies testing of the unfinished code. I’m eager for LSM1 too, but I’m waiting for it to show up in the release notes :) —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
> On Aug 4, 2017, at 9:12 PM, Charles Leifer wrote: > > Right -- my question is still unanswered as to why the code was merged in. It’s not uncommon in software engineering to put unfinished long-running feature work on the main development branch but disable it with a flag. (It was standard policy in one of the projects I worked on at Google.) This helps keep the mainline code and unfinished code from diverging, and it simplifies testing of the unfinished code. I’m eager for LSM1 too, but I’m waiting for it to show up in the release notes :) —Jens ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
On 8/5/17, Charles Leifer wrote: > Right -- my question is still unanswered as to why the code was merged in. The code was merged due to bad planning. We thought LSM1 would be ready in time for 3.20.0, but we underestimated the amount of work involved in getting it there, and also the amount of other distractions that would draw our attention away prior to the 3.20.0 release. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
Right -- my question is still unanswered as to why the code was merged in. I have tremendous respect for this project and everyone involved in it, so please understand I'm not trying to call into question your decisions -- I'd just like clarification as to why this was included now? On Fri, Aug 4, 2017 at 10:48 PM, Richard Hipp wrote: > On 8/4/17, Charles Leifer wrote: > > > > My excitement quickly turned to disappointment as I realized that the > > extension is unusable for all practical purposes: > > > > Yes, it needs work. That is why it is unannounced. > > -- > D. Richard Hipp > d...@sqlite.org > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] LSM1 extension
On 8/4/17, Charles Leifer wrote: > > My excitement quickly turned to disappointment as I realized that the > extension is unusable for all practical purposes: > Yes, it needs work. That is why it is unannounced. -- D. Richard Hipp d...@sqlite.org ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] LSM1 extension
Hi, I was pleased to see that 3.20 contained the source for the LSM1 virtual table, even if not included in the official announcements. My excitement quickly turned to disappointment as I realized that the extension is unusable for all practical purposes: - Single-key equality lookups are broken (unless you hand-craft the encoding of the key according to the encodeKey rules). - Everything else is a full-database scan. I suppose I can understand releasing the code without all the xBestIndex optimizations, but I was really quite surprised to see single-key lookups weren't working as I'd have expected. I don't want to be presumptuous, but I'm having a hard time understanding why this code is included alongside extensions like FTS5 and RTREE. Because it wasn't in the release notes are we to ignore it? That doesn't make sense to me because, given the wonderful tools we have for things like version control and all that, why not just hold off on merging it until it's actually ready for release? Looking forward to hearing your thoughts on the issue, Charlie ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] lsm1 extension
Thanks for the clarification! On Tue, Jun 20, 2017 at 2:41 AM Dan Kennedy wrote: > On 06/20/2017 12:42 PM, Charles Leifer wrote: > > Hi, > > > > I'm not quite sure of the proper way to compile the lsm1 extension (in > the > > lsm-vtab branch). I ended up hand-hacking the makefile to replace $(TCCX) > > and add the appropriate flags for gcc (-fPIC) to get it to build. I'd > like > > to do it the "right way", though, as I was hoping to share some > > instructions on this exciting new extension with a colleague. > > That's how I usually build it. The Makefile in my build directory has: > >TCC = gcc -g -fPIC -Wall -DSQLITE_DEBUG -DSQLITE_NO_SYNC > >... > >include $(TOP)/main.mk >LSMDIR=$(TOP)/ext/lsm1 >include $(LSMDIR)/Makefile > > Then just [make lsm.so]. There is no way to use the configure script to > build lsm1 at the moment. > > Dan. > > > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] lsm1 extension
On 06/20/2017 12:42 PM, Charles Leifer wrote: Hi, I'm not quite sure of the proper way to compile the lsm1 extension (in the lsm-vtab branch). I ended up hand-hacking the makefile to replace $(TCCX) and add the appropriate flags for gcc (-fPIC) to get it to build. I'd like to do it the "right way", though, as I was hoping to share some instructions on this exciting new extension with a colleague. That's how I usually build it. The Makefile in my build directory has: TCC = gcc -g -fPIC -Wall -DSQLITE_DEBUG -DSQLITE_NO_SYNC ... include $(TOP)/main.mk LSMDIR=$(TOP)/ext/lsm1 include $(LSMDIR)/Makefile Then just [make lsm.so]. There is no way to use the configure script to build lsm1 at the moment. Dan. ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] lsm1 extension
Hi, I'm not quite sure of the proper way to compile the lsm1 extension (in the lsm-vtab branch). I ended up hand-hacking the makefile to replace $(TCCX) and add the appropriate flags for gcc (-fPIC) to get it to build. I'd like to do it the "right way", though, as I was hoping to share some instructions on this exciting new extension with a colleague. Anyone able to help? Thanks! Charlie ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users