Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Clint Byrum]

Excerpts from Otto Kekäläinen's message of 2016-07-03 12:46:29 +0100:
> > 3. libmysqlclient.so.18
> 
> Here theres's no concensus yet within the pkg-mysql-maint team on this one.
> 

And IMO, there won't be one until MariaDB accepts that they've created a
forked library that is API-incompatible with libmysqlclient.

> I suggest we extend the mysql-defaults source package to provide a
> real default-mysqlclient-dev metapackage, which other packages can
> build depend on, using versions if needed (just as default-jdk does).
> 
> Current mariadb-10.0 source package does not ship any
> libmariadbclient18 nor libmariadbclient-dev packages at all, as
> previously it was seen as against the policy (but mariadb-5.5 in
> Debian did). I suggest we start producing them now again to make a
> libmysqlclient.so(.18) available from mariadb-10.0.
> 
> Having two libmysqlclient.so.18 files from two different packages is a
> controversial topic. They are not identical so it is ugly to use the
> same name. This is however a necessity dictated by the need to provide
> a drop-in-replacement that can be used directly without going into
> every single clienting program and editing the C headers and soname
> calls.
> 

It's _extremely_ ugly to use this name. MariaDB needs to stop shipping a
library that claims to be libmysqlclient, but is not.

This notion of a drop-in replacement is just a bait-and-switch tactic
(even if it isn't meant to be one). By allowing somebody to build
against a new API without changing their build scripts, it's just
allowing them to accidentally use that new API and then unknowingly be
dependent on symbols only available in the forked library. There's a
different between ABI equality, and ABI compatibility, and IMO, the
former is more important for Debian.

> It should be noted that the Debian packages shipped directly from
> mariadb.org have provided the libmysqlclient.so file since forever and
> there has not been problems and the drop-in-replacement function has
> been fullfilled as expected. All packages that used to depend on
> Oracle MySQL client library have continued to function with the
> MariaDB equivalent when mariadb.org repositories are activated.
> 

There have not been problems that users reported to MariaDB, because
the issue is extremely subtle and subversive. Those users would only
ever know that there was a problem if they wanted to distribute their
code and had another user try to build on libmysqlclient. For the
average MariaDB user, that's not such a big deal, as they may not be
distributing their code. For Debian, we're distributing all of it, and
we have a duty to give users reliable software. We don't want to build
against libmariadbclient-dev-compat and then the software just doesn't
work with libmysqlclient.so.18 from MySQL.

> It should also be noted that the soname version number 18 has been
> used in Oracle MySQL for a long time without bumping it despite
> changes in the API. The API changes have also gone undocumented all
> the time as the libmysqlclient18 package does not have .symbols file.
> Oracle has finally bumped the soname version to 20 in MySQL 5.7. The
> number 19 is not used anywhere, but was left as something that can be
> resorted to in 5.6, in case somebody would complain too much that 5.5
> and 5.6 API differ but have same version number. No one has, so for
> all practical purposes, the current situation is OK.
> 

The past was horrible, and MySQL obviously did weird things and our
packages were not sufficient to police this properly. But that doesn't
change the fact that MariaDB has hijacked the mysqlcient soname.

> The solution I suggest works nicely in the scope of
> libmysqlclient18/libmariadbclient18 and is resilent for scenarios
> where libmysqlclient20 is introduced or where even more client
> libraries are available (mainly libmariadb2), but I won't go into the
> details of those now, as I think here is enough information to decide
> on whether or not it is OK to provide libmysqlclient.so from a
> libmariadbclient18 package. This would also mean we introduce the
> default-libmysqlclient-dev pmetaackage which depends on either
> libmariadbclient-dev or libmysqlclient-dev package.
> 
> Please comment and advice on how to proceed with packaging to satisfy
> with the switchable defaults requirement.
>

IMO, it's only OK for MariaDB to provide libmysqlclient.so if it is 100%
equivalent to MySQL's libmysqlclient.so. If there are extra API calls,
and extra symbols, it's not the same, and it won't produce a compatible
program linked to libmysqlclient.so.18.

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Andreas Beckmann]

On 2016-07-06 09:54, Otto Kekäläinen wrote:
> 2016-07-05 19:52 GMT+03:00 Robie Basak :
>> PROPOSAL B
> 
> The whole pkg-mysql-maint team is behind the proposal Robie now wrote
> about. We have been discussing actively and engineering it during the
> last two days. It was also one of the topics discussed at the
> MariaDB/MySQL Bof at DebConf yesterday and we agreed about it. Only
> minor implementation details are left to decide on, like what command
> to use to make the links multiarch.

Nice! Can you summarize what what the new real/virtual packages will be
called and what packages are supposed to put into their
Depends/Build-Depends? (General packages that can work with any vendor
as well as packages specific to a single vendor.)

I see that the new unified repository is on alioth for mysql-5.7 and
mysql-defaults. I've now updated my repo at
git+ssh://git.debian.org/git/users/anbe/tmp/mysql.git
with a proposed integration of mysql-5.6, too.
There are also branches for dropping the mysql-common package from both
mysql-5.6 and mysql-5.7 once mysql-defaults is uploaded.

> If you want to have a say, please do so now before we finalize the
> solution we currently believe is the best one.
> 
> I expect to upload this new scheme to experimental in a few days for a
> larger audience to test and verify.


Andreas

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Otto Kekäläinen]

2016-07-05 19:52 GMT+03:00 Robie Basak :
> PROPOSAL B

The whole pkg-mysql-maint team is behind the proposal Robie now wrote
about. We have been discussing actively and engineering it during the
last two days. It was also one of the topics discussed at the
MariaDB/MySQL Bof at DebConf yesterday and we agreed about it. Only
minor implementation details are left to decide on, like what command
to use to make the links multiarch.

If you want to have a say, please do so now before we finalize the
solution we currently believe is the best one.

I expect to upload this new scheme to experimental in a few days for a
larger audience to test and verify.

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Robie Basak]

On Tue, Jul 05, 2016 at 11:41:15AM +0200, Jonathan Wiltshire wrote:
> > I suggest we extend the mysql-defaults source package to provide a
> > real default-mysqlclient-dev metapackage, which other packages can
> > build depend on, using versions if needed (just as default-jdk does).
> 
> Yes.
> 
> > Current mariadb-10.0 source package does not ship any
> > libmariadbclient18 nor libmariadbclient-dev packages at all, as
> > previously it was seen as against the policy (but mariadb-5.5 in
> > Debian did). I suggest we start producing them now again to make a
> > libmysqlclient.so(.18) available from mariadb-10.0.
> > 
> > Having two libmysqlclient.so.18 files from two different packages is a
> > controversial topic. They are not identical so it is ugly to use the
> > same name. This is however a necessity dictated by the need to provide
> > a drop-in-replacement that can be used directly without going into
> > every single clienting program and editing the C headers and soname
> > calls.
> 
> We discussed this in the release team and concluded that having two
> libmysqlclient.so.18 files in two packages is likely to run into problems
> later if/when they diverge, so there is no point delaying that pain.
> Especially as they are not identical even now.

Sorry, I'm not clear on the sense of your statement. By this, are you
agreeing to do it now, or saying that we shouldn't do it?

> I wonder if pkgconfig can be any help here? That involves a one-time change
> to client packages if they don't already use pkgconfig but doesn't have to
> be repeated if the default switches.

I experimented down another route. In my proposal, I keep the
libmysqlclient for MySQL only, and use libmariadbclient for the MariaDB
upstream. So:

PROPOSAL B

pkg:libmysqlclient18 ships libmysqlclient.so.18
pkg:libmysqlclient-dev ships libmysqlclient.so

(no change so far).

Then:

pkg:libmariadbclient18 is created to ship libmariadbclient.so.18
pkg:libmariadbclient-dev is created to ship libmariadbclient.so

These are new additions to src:mariadb-10.0. We have to patch a little
to switch from MariaDB from building libmysqlclient to use
libmariadbclient instead.

pkg:libmariadbclient-dev must conflict with pkg:libmysqlclient-dev
because they ship the same header files, but this affects builds only,
and not (normal) runtime.

Then:

pkg:libmariadbclient-dev-compat is created to ship a symlink:
libmysqlclient.so -> libmariadbclient.so. This must conflict with
pkg:libmysqlclient-dev, but again we're affecting builds only, and not
normal runtime.

pkg:default-libmysqlclient-dev is created to depend on
pkg:libmariadbclient-dev-compat.

What this gets us:

 1) Maintainers can build against whichever one they prefer, and/or the
 release team is in a position to be able to mandate a default as
 required.

 2) Users can build against either, and continue to use either. There is
 no system wide choice being forced here.

 3) No confusion between MySQL and MariaDB. "libmysqlclient.so.18" is
 always MySQL and "libmariadbclient.so.18" is always MariaDB.

 4) Maintainers need only change their build dependencies. Upstreams
 that try to link using "-lmysqlclient" will continue to work.

END OF PROPOSAL


I tested this and it appears to work.
https://git.launchpad.net/~racb/ubuntu/+source/mariadb-10.0/log/ is my
MariaDB test modification (PoC with amd64 hardcoded). I created a
fake "equivs" libmysqlclient-dev that depends only on
libmariadbclient-dev-compat, and dropped the necessary conflicts, for
testing purposes. cacti-spine (a simple consumer of libmysqlclient) then
successfully linked against libmariadbclient.so.18 and depended on
pkg:libmariadbclient18 with no further modification.

If we want to test more packages, it's sufficient to just rebuild them
with my test build dependencies available.

Some commentary on my justifications above:

1) I believe this is the core of what the release team requested in
terms of MySQL and MariaDB. I appreciate of course that whatever hackery
we do to make this happen is something the release team should weigh in
on also.

3) I really don't want to go down the route of "libmysqlclient.so.18"
actually being MariaDB. It may be ABI-compatible now, but what about the
future? MySQL 5.7 already bumps to libmysqlclient.so.20, for example. If
and until MariaDB ships an ABI-compatible libmysqlclient.so.20, are we
going to ship both, one from each upstream? What if MariaDB's one turns
out not to be ABI compatible? I think down this path lies madness.
sonames give us a method to avoid this conflict by using different
names. We should use this if we can.

Feedback on this approach appreciated.

Robie

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[peter green]

2016-07-02 17:53 GMT+03:00 Otto Kekäläinen:
>  This is modeled after how default-jdk or default-mta works.

Actually default-jdk is a real metapackage, while default-mta is a
pure virtual package. I am not sure which way to pick here now.

https://packages.debian.org/jessie/default-mta
https://packages.debian.org/jessie/default-jdk
   

It depends on what behaviour you want on dist-upgrade.

A real package will stick arround, so if you switch the default then a 
dist-upgrade will pull in the new default. On the other hand a virtual 
package is never installed as such, so existing systems will keep their 
existing implementation.

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Otto Kekäläinen]

2016-07-02 17:53 GMT+03:00 Otto Kekäläinen :
> This is modeled after how default-jdk or default-mta works.

Actually default-jdk is a real metapackage, while default-mta is a
pure virtual package. I am not sure which way to pick here now.

https://packages.debian.org/jessie/default-mta
https://packages.debian.org/jessie/default-jdk

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Otto Kekäläinen]

Hello!

What does the release team think of introducing new purely virtual
packages named default-mysq-*, which are provided only by mariadb-*
packages?
This is modeled after how default-jdk or default-mta works.

In code if would look like this:
https://github.com/ottok/mariadb-10.0/commit/e162c743fcd59273ba948e19b71cb5ed5638f0f5

I am at DebConf. Grab me by the sleeve when you see me. I'd like to
have external input/advice on this and a in-person discussion about
the scenarios and deployment path would feel helpful to me.

- Otto

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Emilio Pozuelo Monfort]

On 24/06/16 20:05, Robie Basak wrote:
> On Fri, Jun 24, 2016 at 07:53:56PM +0200, Julien Cristau wrote:
>> Please don't use virtual-foo for non-virtual packages, that way lies
>> madness and confusion and despair :)
> 
> OK, but does anyone have any objection to the principle of my
> suggestion, even if we must use a different name?

You could do something like that. Or something like what we do with default-mta
and mail-transport-agent.

Cheers,
Emilio

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Robie Basak]

On Fri, Jun 24, 2016 at 07:53:56PM +0200, Julien Cristau wrote:
> Please don't use virtual-foo for non-virtual packages, that way lies
> madness and confusion and despair :)

OK, but does anyone have any objection to the principle of my
suggestion, even if we must use a different name?

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Julien Cristau]

On Fri, Jun 24, 2016 at 14:55:34 +0100, Robie Basak wrote:

> On Wed, Jun 01, 2016 at 07:03:42PM +0200, Andreas Beckmann wrote:
> > I prepared a prototype for a separate mysql-common package (and
> > default-* packages) at
> > git://git.debian.org/git/users/anbe/tmp/mysql-defaults.git
> > but so far nobody had time to review it.
> 
> A thought from Norvald. Do we really need more metapackages? What if we
> converted the existing virtual-mysql-{server-client} virtual packages
> into metapackages generated from src:mysql-defaults instead?
> 
> Then these would depend on "mariadb-server | mysql-server" etc. Existing
> packages would drop their preferred alternative and depend on
> "virtual-mysql-server" only. src:mysql-defaults would then have full
> control of the "default" just by swapping the alternatives.
> 
> The benefit would be fewer meta/virtual -packages. Is there any use case
> that is not covered by this?

Please don't use virtual-foo for non-virtual packages, that way lies
madness and confusion and despair :)

Cheers,
Julien


signature.asc
Description: PGP signature

Re: [debian-mysql] About packages that depend on mysql-* / mariadb / virtual-mysql-*

[Robie Basak]

On Wed, Jun 01, 2016 at 07:03:42PM +0200, Andreas Beckmann wrote:
> I prepared a prototype for a separate mysql-common package (and
> default-* packages) at
> git://git.debian.org/git/users/anbe/tmp/mysql-defaults.git
> but so far nobody had time to review it.

A thought from Norvald. Do we really need more metapackages? What if we
converted the existing virtual-mysql-{server-client} virtual packages
into metapackages generated from src:mysql-defaults instead?

Then these would depend on "mariadb-server | mysql-server" etc. Existing
packages would drop their preferred alternative and depend on
"virtual-mysql-server" only. src:mysql-defaults would then have full
control of the "default" just by swapping the alternatives.

The benefit would be fewer meta/virtual -packages. Is there any use case
that is not covered by this?


signature.asc
Description: PGP signature