Re: [sqlite] LSM1 extension

[Richard Hipp]

On 8/9/17, Jens Alfke  wrote:
>
> One thing I’m unclear on: Will the LSM1 extension in SQLite3 be able to plug
> in at a low level (replacing the b-tree engine), so one can use it with all
> the existing features like relational tables and SQL queries; or will it
> provide only a basic key/value store API?

It is just a separate key/value store using a different file.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[Jens Alfke]

> On Aug 9, 2017, at 8:23 AM, Nico Williams  wrote:
> 
> I must say though, SQLite4 is *brilliant*.  I do hope that it either
> gets completed and replaces SQLite3, or has all its good ideas folded
> into SQLite3.  The LSM1 work in SQLite3 seems to point towords the
> latter.

One thing I’m unclear on: Will the LSM1 extension in SQLite3 be able to plug in 
at a low level (replacing the b-tree engine), so one can use it with all the 
existing features like relational tables and SQL queries; or will it provide 
only a basic key/value store API?

(Sorry if it’s off-topic to discuss future features here. I know there’s a 
development-focused list but I think I’d be way beyond my depth there. LSM1 is 
potentially very, very relevant to the project I’m working on, but that depends 
on how it ends up integrated, which is why I ask.)

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[Nico Williams]

On Wed, Aug 09, 2017 at 08:26:51AM -0500, Charles Leifer wrote:
> sqlite4's future is uncertain. It's not, as I understand, meant to replace
> sqlite3 any time soon. I think it was more of a place to try out new ideas
> for implementations.

I must say though, SQLite4 is *brilliant*.  I do hope that it either
gets completed and replaces SQLite3, or has all its good ideas folded
into SQLite3.  The LSM1 work in SQLite3 seems to point towords the
latter.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[Charles Leifer]

sqlite4's future is uncertain. It's not, as I understand, meant to replace
sqlite3 any time soon. I think it was more of a place to try out new ideas
for implementations.

On Wed, Aug 9, 2017 at 3:10 AM, x <tam118...@hotmail.com> wrote:

> Thanks Charles. Is sqlite4 available yet?
>
> From: Charles Leifer<mailto:colei...@gmail.com>
> Sent: 08 August 2017 21:21
> To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org>
> Subject: Re: [sqlite] LSM1 extension
>
> There's some information that may be of interest on the sqlite4 wiki:
>
> * Design overview: https://sqlite.org/src4/doc/trunk/www/lsm.wiki
> * User guide: https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki
> * API: https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki
>
> My interest in this particular feature stems from my having written a set
> of python bindings to the sqlite4 implementation (to be used as a sort of
> embedded ordered key/value storage engine). I've wondered about the
> possibility of exposing these APIs as a virtual table, and so seeing this
> extension really excited me. So obviously my interest is much greater than
> that of the average SQLite user :)
>
> On Tue, Aug 8, 2017 at 12:54 PM, x <tam118...@hotmail.com> wrote:
>
> > Thanks Richard.
> >
> > From: Richard Hipp<mailto:d...@sqlite.org>
> > Sent: 08 August 2017 18:47
> > To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org>
> > Subject: Re: [sqlite] LSM1 extension
> >
> > On 8/8/17, x <tam118...@hotmail.com> wrote:
> > > I keep reading about this LSM1 but can’t find anything about it beyond
> > > tickets. Any links?
> >
> > Baseline technology: https://en.wikipedia.org/wiki/
> > Log-structured_merge-tree
> >
> > No documentation yet on the LSM1 implementation in SQLite.
> >
> > --
> > D. Richard Hipp
> > d...@sqlite.org
> > ___
> > sqlite-users mailing list
> > sqlite-users@mailinglists.sqlite.org
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> >
> > ___
> > sqlite-users mailing list
> > sqlite-users@mailinglists.sqlite.org
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> >
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[x]

Thanks Charles. Is sqlite4 available yet?

From: Charles Leifer<mailto:colei...@gmail.com>
Sent: 08 August 2017 21:21
To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org>
Subject: Re: [sqlite] LSM1 extension

There's some information that may be of interest on the sqlite4 wiki:

* Design overview: https://sqlite.org/src4/doc/trunk/www/lsm.wiki
* User guide: https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki
* API: https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki

My interest in this particular feature stems from my having written a set
of python bindings to the sqlite4 implementation (to be used as a sort of
embedded ordered key/value storage engine). I've wondered about the
possibility of exposing these APIs as a virtual table, and so seeing this
extension really excited me. So obviously my interest is much greater than
that of the average SQLite user :)

On Tue, Aug 8, 2017 at 12:54 PM, x <tam118...@hotmail.com> wrote:

> Thanks Richard.
>
> From: Richard Hipp<mailto:d...@sqlite.org>
> Sent: 08 August 2017 18:47
> To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org>
> Subject: Re: [sqlite] LSM1 extension
>
> On 8/8/17, x <tam118...@hotmail.com> wrote:
> > I keep reading about this LSM1 but can’t find anything about it beyond
> > tickets. Any links?
>
> Baseline technology: https://en.wikipedia.org/wiki/
> Log-structured_merge-tree
>
> No documentation yet on the LSM1 implementation in SQLite.
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[Charles Leifer]

There's some information that may be of interest on the sqlite4 wiki:

* Design overview: https://sqlite.org/src4/doc/trunk/www/lsm.wiki
* User guide: https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki
* API: https://sqlite.org/src4/doc/trunk/www/lsmapi.wiki

My interest in this particular feature stems from my having written a set
of python bindings to the sqlite4 implementation (to be used as a sort of
embedded ordered key/value storage engine). I've wondered about the
possibility of exposing these APIs as a virtual table, and so seeing this
extension really excited me. So obviously my interest is much greater than
that of the average SQLite user :)

On Tue, Aug 8, 2017 at 12:54 PM, x <tam118...@hotmail.com> wrote:

> Thanks Richard.
>
> From: Richard Hipp<mailto:d...@sqlite.org>
> Sent: 08 August 2017 18:47
> To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org>
> Subject: Re: [sqlite] LSM1 extension
>
> On 8/8/17, x <tam118...@hotmail.com> wrote:
> > I keep reading about this LSM1 but can’t find anything about it beyond
> > tickets. Any links?
>
> Baseline technology: https://en.wikipedia.org/wiki/
> Log-structured_merge-tree
>
> No documentation yet on the LSM1 implementation in SQLite.
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[x]

Thanks Richard.

From: Richard Hipp<mailto:d...@sqlite.org>
Sent: 08 August 2017 18:47
To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org>
Subject: Re: [sqlite] LSM1 extension

On 8/8/17, x <tam118...@hotmail.com> wrote:
> I keep reading about this LSM1 but can’t find anything about it beyond
> tickets. Any links?

Baseline technology: https://en.wikipedia.org/wiki/Log-structured_merge-tree

No documentation yet on the LSM1 implementation in SQLite.

--
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[Richard Hipp]

On 8/8/17, x  wrote:
> I keep reading about this LSM1 but can’t find anything about it beyond
> tickets. Any links?

Baseline technology: https://en.wikipedia.org/wiki/Log-structured_merge-tree

No documentation yet on the LSM1 implementation in SQLite.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[x]

I keep reading about this LSM1 but can’t find anything about it beyond tickets. 
Any links?

From: Jens Alfke<mailto:j...@mooseyard.com>
Sent: 08 August 2017 18:04
To: SQLite mailing list<mailto:sqlite-users@mailinglists.sqlite.org>
Subject: Re: [sqlite] LSM1 extension


> On Aug 4, 2017, at 9:12 PM, Charles Leifer <colei...@gmail.com> wrote:
>
> Right -- my question is still unanswered as to why the code was merged in.

It’s not uncommon in software engineering to put unfinished long-running 
feature work on the main development branch but disable it with a flag. (It was 
standard policy in one of the projects I worked on at Google.) This helps keep 
the mainline code and unfinished code from diverging, and it simplifies testing 
of the unfinished code.

I’m eager for LSM1 too, but I’m waiting for it to show up in the release notes 
:)

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[Jens Alfke]

> On Aug 4, 2017, at 9:12 PM, Charles Leifer  wrote:
> 
> Right -- my question is still unanswered as to why the code was merged in.

It’s not uncommon in software engineering to put unfinished long-running 
feature work on the main development branch but disable it with a flag. (It was 
standard policy in one of the projects I worked on at Google.) This helps keep 
the mainline code and unfinished code from diverging, and it simplifies testing 
of the unfinished code.

I’m eager for LSM1 too, but I’m waiting for it to show up in the release notes 
:)

—Jens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[Richard Hipp]

On 8/5/17, Charles Leifer  wrote:
> Right -- my question is still unanswered as to why the code was merged in.

The code was merged due to bad planning.  We thought LSM1 would be
ready in time for 3.20.0, but we underestimated the amount of work
involved in getting it there, and also the amount of other
distractions that would draw our attention away prior to the 3.20.0
release.
-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[Charles Leifer]

Right -- my question is still unanswered as to why the code was merged in.
I have tremendous respect for this project and everyone involved in it, so
please understand I'm not trying to call into question your decisions --
I'd just like clarification as to why this was included now?

On Fri, Aug 4, 2017 at 10:48 PM, Richard Hipp  wrote:

> On 8/4/17, Charles Leifer  wrote:
> >
> > My excitement quickly turned to disappointment as I realized that the
> > extension is unusable for all practical purposes:
> >
>
> Yes, it needs work.  That is why it is unannounced.
>
> --
> D. Richard Hipp
> d...@sqlite.org
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] LSM1 extension

[Richard Hipp]

On 8/4/17, Charles Leifer  wrote:
>
> My excitement quickly turned to disappointment as I realized that the
> extension is unusable for all practical purposes:
>

Yes, it needs work.  That is why it is unannounced.

-- 
D. Richard Hipp
d...@sqlite.org
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] LSM1 extension

[Charles Leifer]

Hi,

I was pleased to see that 3.20 contained the source for the LSM1 virtual
table, even if not included in the official announcements.

My excitement quickly turned to disappointment as I realized that the
extension is unusable for all practical purposes:

- Single-key equality lookups are broken (unless you hand-craft the
encoding of the key according to the encodeKey rules).
- Everything else is a full-database scan.

I suppose I can understand releasing the code without all the xBestIndex
optimizations, but I was really quite surprised to see single-key lookups
weren't working as I'd have expected.

I don't want to be presumptuous, but I'm having a hard time understanding
why this code is included alongside extensions like FTS5 and RTREE. Because
it wasn't in the release notes are we to ignore it? That doesn't make sense
to me because, given the wonderful tools we have for things like version
control and all that, why not just hold off on merging it until it's
actually ready for release?

Looking forward to hearing your thoughts on the issue,

Charlie
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] lsm1 extension

[Charles Leifer]

Thanks for the clarification!

On Tue, Jun 20, 2017 at 2:41 AM Dan Kennedy  wrote:

> On 06/20/2017 12:42 PM, Charles Leifer wrote:
> > Hi,
> >
> > I'm not quite sure of the proper way to compile the lsm1 extension (in
> the
> > lsm-vtab branch). I ended up hand-hacking the makefile to replace $(TCCX)
> > and add the appropriate flags for gcc (-fPIC) to get it to build. I'd
> like
> > to do it the "right way", though, as I was hoping to share some
> > instructions on this exciting new extension with a colleague.
>
> That's how I usually build it. The Makefile in my build directory has:
>
>TCC = gcc -g -fPIC -Wall -DSQLITE_DEBUG -DSQLITE_NO_SYNC
>
>...
>
>include $(TOP)/main.mk
>LSMDIR=$(TOP)/ext/lsm1
>include $(LSMDIR)/Makefile
>
> Then just [make lsm.so]. There is no way to use the configure script to
> build lsm1 at the moment.
>
> Dan.
>
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] lsm1 extension

[Dan Kennedy]

On 06/20/2017 12:42 PM, Charles Leifer wrote:

Hi,

I'm not quite sure of the proper way to compile the lsm1 extension (in the
lsm-vtab branch). I ended up hand-hacking the makefile to replace $(TCCX)
and add the appropriate flags for gcc (-fPIC) to get it to build. I'd like
to do it the "right way", though, as I was hoping to share some
instructions on this exciting new extension with a colleague.


That's how I usually build it. The Makefile in my build directory has:

  TCC = gcc -g -fPIC -Wall -DSQLITE_DEBUG -DSQLITE_NO_SYNC

  ...

  include $(TOP)/main.mk
  LSMDIR=$(TOP)/ext/lsm1
  include $(LSMDIR)/Makefile

Then just [make lsm.so]. There is no way to use the configure script to 
build lsm1 at the moment.


Dan.


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] lsm1 extension

[Charles Leifer]

Hi,

I'm not quite sure of the proper way to compile the lsm1 extension (in the
lsm-vtab branch). I ended up hand-hacking the makefile to replace $(TCCX)
and add the appropriate flags for gcc (-fPIC) to get it to build. I'd like
to do it the "right way", though, as I was hoping to share some
instructions on this exciting new extension with a colleague.

Anyone able to help? Thanks!

Charlie
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users