Re: Using ndb in RPM (was: Re: F27 System Wide Change: RPM 4.14)

[Richard W.M. Jones]

On Mon, Jul 10, 2017 at 11:28:13AM +0200, Jos Vos wrote:
> Hi,
> 
> On Mon, Jul 10, 2017 at 09:14:07AM +0100, Richard W.M. Jones wrote:
> 
> > The web page has:
> > 
> >Improvements and stabilization of "ndb" (New RPM DB Format database
> >format)
> >
> > [...]
> > 
> > The problem is that "NDB" is a custom homebrew database invented in
> > the RPM codebase.  I agree that because of licensing problems we need
> > to move away from Berkeley DB, but why not switch to the obvious,
> > bulletproof choice - Sqlite?
> 
> SQLite is is totally different piece of cake (RDBMS vs. key/value).
> 
> Why not use LMDB?  It's the replacement for BerkeleyDB in OpenLDAP.
> 
> http://www.lmdb.tech/doc/
> 
> It's extremely fast...

TBH I'm not fussed about what specific technology is used as long as
it's not some homebrew thing, is battle-tested, and is reasonably
widely available with as few external dependencies as possible.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Using ndb in RPM (was: Re: F27 System Wide Change: RPM 4.14)

[Neal Gompa]

On Mon, Jul 10, 2017 at 4:14 AM, Richard W.M. Jones  wrote:
> On Fri, Jul 07, 2017 at 04:15:53PM +0200, Jaroslav Reznik wrote:
>> = System Wide Change: RPM 4.14 =
>> https://fedoraproject.org/wiki/Changes/RPM-4.14
>
> The web page has:
>
>Improvements and stabilization of "ndb" (New RPM DB Format database
>format)
> ...
>Changing database from bdb to ndb requires patching of some programs
>who read/expect /var/lib/rpm/Packages and other files directly
>without using librpm
>
> which were not included in this email.
>
> The problem is that "NDB" is a custom homebrew database invented in
> the RPM codebase.  I agree that because of licensing problems we need
> to move away from Berkeley DB, but why not switch to the obvious,
> bulletproof choice - Sqlite?
>
> Look at the all new nbd code here:
>
>   https://github.com/rpm-software-management/rpm/tree/master/lib/backend/ndb
>
> Inventing a new database including a new disk format and all new code
> is bound to cause problems with database corruption, incorrect
> database synchronization, incorrect handling of parallel queries, and
> a rich source of other bugs for a long time.
>
> Not to mention ‘ndb requires patching of some programs who read/expect
> /var/lib/rpm/Packages and other files directly without using librpm’ -
> this would still require changes, but they are much smaller if you're
> using sqlite.
>
> It was a bad upstream decision, but it's not too late to avoid it by
> sticking with BDB for now and using sqlite in the long run.
>

A long time ago, we did actually have a SQLite backend. It was removed
in RPM 4.9.0[1][2]. I'm not sure *why* it was removed.

[1]: http://rpm.org/wiki/Releases/4.9.0
[2]: 
https://github.com/rpm-software-management/rpm/commit/e2c217b4b76118e6dab9f8dfb3284bb4ddbe2b3e

On Mon, Jul 10, 2017 at 5:28 AM, Jos Vos  wrote:
> Hi,
>
> On Mon, Jul 10, 2017 at 09:14:07AM +0100, Richard W.M. Jones wrote:
>
>> The web page has:
>>
>>Improvements and stabilization of "ndb" (New RPM DB Format database
>>format)
>>
>> [...]
>>
>> The problem is that "NDB" is a custom homebrew database invented in
>> the RPM codebase.  I agree that because of licensing problems we need
>> to move away from Berkeley DB, but why not switch to the obvious,
>> bulletproof choice - Sqlite?
>
> SQLite is is totally different piece of cake (RDBMS vs. key/value).
>
> Why not use LMDB?  It's the replacement for BerkeleyDB in OpenLDAP.
>
> http://www.lmdb.tech/doc/
>
> It's extremely fast...
>

LMDB has been brought up in the past[3] as well as recently[4].

[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1086784
[4]: https://github.com/rpm-software-management/rpm/issues/128


-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Using ndb in RPM (was: Re: F27 System Wide Change: RPM 4.14)

[Jos Vos]

Hi,

On Mon, Jul 10, 2017 at 09:14:07AM +0100, Richard W.M. Jones wrote:

> The web page has:
> 
>Improvements and stabilization of "ndb" (New RPM DB Format database
>format)
>
> [...]
> 
> The problem is that "NDB" is a custom homebrew database invented in
> the RPM codebase.  I agree that because of licensing problems we need
> to move away from Berkeley DB, but why not switch to the obvious,
> bulletproof choice - Sqlite?

SQLite is is totally different piece of cake (RDBMS vs. key/value).

Why not use LMDB?  It's the replacement for BerkeleyDB in OpenLDAP.

http://www.lmdb.tech/doc/

It's extremely fast...

Cheers,

-- 
--Jos Vos 
--X/OS Experts in Open Systems BV   |   Office: +31 20 6938364
--Amsterdam, The Netherlands|   Mobile: +31 6 26216181
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Using ndb in RPM (was: Re: F27 System Wide Change: RPM 4.14)

[Richard W.M. Jones]

On Fri, Jul 07, 2017 at 04:15:53PM +0200, Jaroslav Reznik wrote:
> = System Wide Change: RPM 4.14 =
> https://fedoraproject.org/wiki/Changes/RPM-4.14

The web page has:

   Improvements and stabilization of "ndb" (New RPM DB Format database
   format)
...
   Changing database from bdb to ndb requires patching of some programs
   who read/expect /var/lib/rpm/Packages and other files directly
   without using librpm

which were not included in this email.

The problem is that "NDB" is a custom homebrew database invented in
the RPM codebase.  I agree that because of licensing problems we need
to move away from Berkeley DB, but why not switch to the obvious,
bulletproof choice - Sqlite?

Look at the all new nbd code here:

  https://github.com/rpm-software-management/rpm/tree/master/lib/backend/ndb

Inventing a new database including a new disk format and all new code
is bound to cause problems with database corruption, incorrect
database synchronization, incorrect handling of parallel queries, and
a rich source of other bugs for a long time.

Not to mention ‘ndb requires patching of some programs who read/expect
/var/lib/rpm/Packages and other files directly without using librpm’ -
this would still require changes, but they are much smaller if you're
using sqlite.

It was a bad upstream decision, but it's not too late to avoid it by
sticking with BDB for now and using sqlite in the long run.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org