Re: Do opaque struct changes break C library ABIs

[Guillem Jover]

Hi!

On Thu, 2013-05-16 at 17:35:10 -0700, Russ Allbery wrote:
> I have a C shared library that takes a pointer to an opaque struct as the
> first argument to most of its API calls.  The internal layout of that
> opaque struct is changing (to add new members).  The only way to create
> the opaque state struct is via a call to remctl_new(), which returns a
> pointer to it.  Nothing else about the ABI is changing, and client
> programs cannot see or manipulate the struct members directly (at least
> without poking about in memory, of course).
> 
> Is that an ABI break that requires an SONAME change?

Nope, that's the customary way to preserve ABI.

> If you have two versions of the library loaded in memory at the same time,
> call remctl_new() from one version, and then call functions from the other
> version, this will break horribly, since the struct layout has changed and
> will be misinterpreted.  However, all the ABI calls are identical between
> the two versions of the library, so this would require the dynamic linker
> resolve two calls to the same library differently.  Also, if both
> libraries have the same SONAME, I don't see any way that you could load
> two versions into memory at the same time without playing tricks with
> dlopen() that I don't see any reason to support.
> 
> Is that something that one has to guard against, or can one assume that
> the ABI calls from a client of a shared library will be consistent in this
> situation: that all the symbols will resolve to the same version of the
> shared library when both share the same SONAME?

Yeah that should be the case, the dynamic linker should not be
loading the same SONAME multiple times, so there should be no race
here, and I agree dlopen()ing shared libraries in general should not
be supported (I'd even go further and say this should be outright
banned, given the pain it causes, and optional library support should
always be implemented by loading a plugin properly linked against such
library with an ABI under the control of the program loading it).

> I'm pretty sure this change is not an ABI break, but the more I thought
> about it, the more I wasn't absolutely certain that I had the right mental
> model here, so I thought I'd ask.

It could be problematic in case there's more than one instance of the
library with different SONAMEs used by other stuff and none use
versioned symbols, but that's not the case you were presenting here so...

Thanks,
Guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20130517051748.ga6...@gaara.hadrons.org

Re: Do opaque struct changes break C library ABIs

[Josselin Mouette]

Le jeudi 16 mai 2013 à 17:35 -0700, Russ Allbery a écrit : 
> I have a C shared library that takes a pointer to an opaque struct as the
> first argument to most of its API calls.  The internal layout of that
> opaque struct is changing (to add new members).  The only way to create
> the opaque state struct is via a call to remctl_new(), which returns a
> pointer to it.  Nothing else about the ABI is changing, and client
> programs cannot see or manipulate the struct members directly (at least
> without poking about in memory, of course).
> 
> Is that an ABI break that requires an SONAME change?

Opaque structures are usually here precisely to avoid SONAME changes.
Glib and GTK+ do that all the time, and I don’t see applications
breaking horribly, unless they do unwarranted assumptions about the
internals of those structures.

So if this is the only change, I’d say you are pretty safe here.

Cheers,
-- 
 .''`.  Josselin Mouette
: :' :
`. `'
  `-


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1368768298.23162.224.camel@pi0307572

Re: Do opaque struct changes break C library ABIs

[Mathieu Malaterre]

On Fri, May 17, 2013 at 2:35 AM, Russ Allbery  wrote:
> I have a C shared library that takes a pointer to an opaque struct as the
> first argument to most of its API calls.  The internal layout of that
> opaque struct is changing (to add new members).  The only way to create
> the opaque state struct is via a call to remctl_new(), which returns a
> pointer to it.  Nothing else about the ABI is changing, and client
> programs cannot see or manipulate the struct members directly (at least
> without poking about in memory, of course).
>
> Is that an ABI break that requires an SONAME change?

No, and in case previous answers were not convincing enough, you can
still give a-c-c a try:

http://packages.qa.debian.org/a/abi-compliance-checker.html

2cts


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
http://lists.debian.org/CA+7wUsxumTH25j=Uwf=4ckdxpe1s93y8fbgjkzququv9q7_...@mail.gmail.com

Re: Do opaque struct changes break C library ABIs

[Russ Allbery]

Josselin Mouette  writes:
> Le jeudi 16 mai 2013 à 17:35 -0700, Russ Allbery a écrit : 

>> I have a C shared library that takes a pointer to an opaque struct as
>> the first argument to most of its API calls.  The internal layout of
>> that opaque struct is changing (to add new members).  The only way to
>> create the opaque state struct is via a call to remctl_new(), which
>> returns a pointer to it.  Nothing else about the ABI is changing, and
>> client programs cannot see or manipulate the struct members directly
>> (at least without poking about in memory, of course).
>> 
>> Is that an ABI break that requires an SONAME change?

> Opaque structures are usually here precisely to avoid SONAME changes.
> Glib and GTK+ do that all the time, and I don’t see applications
> breaking horribly, unless they do unwarranted assumptions about the
> internals of those structures.

> So if this is the only change, I’d say you are pretty safe here.

Cool, thanks (and to everyone else as well) for the replies.  I was
*fairly* sure that I understood this, but I didn't want to release
something and then be wrong.  :)

-- 
Russ Allbery (r...@debian.org)   


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87r4h6uome@windlord.stanford.edu

Re: Do opaque struct changes break C library ABIs

[Thomas Preud'homme]

Le vendredi 17 mai 2013 07:17:48, Guillem Jover a écrit :
> (…) I agree dlopen()ing shared libraries in general should not
> be supported (I'd even go further and say this should be outright
> banned, given the pain it causes, and optional library support should
> always be implemented by loading a plugin properly linked against such
> library with an ABI under the control of the program loading it).

I know of at least one case where I think it makes sense. TCC compiler uses 
dlopen to run a C file as a script. It compiles to memory and then dlopen all 
the libraries it needs and resolve the symbol itself. If the library in 
question is also used by the compiler (eg. glibc), it's going to be opened 
twice.

However the dlopening is done shortly after the compiler is run so it's very 
unlikely that the dlopened version of the library is different than the one 
loaded by the dynamic linker so I don't think it's a problem in practice.

Surely such a case shouldn't be banned.


signature.asc
Description: This is a digitally signed message part.

Re: Do opaque struct changes break C library ABIs

[Chow Loong Jin]

On 17/05/2013 13:17, Guillem Jover wrote:
> Yeah that should be the case, the dynamic linker should not be
> loading the same SONAME multiple times, so there should be no race
> here, and I agree dlopen()ing shared libraries in general should not
> be supported (I'd even go further and say this should be outright
> banned, given the pain it causes, and optional library support should
> always be implemented by loading a plugin properly linked against such
> library with an ABI under the control of the program loading it).

But how do you load a plugin without using dlopen()?

-- 
Kind regards,
Loong Jin



signature.asc
Description: OpenPGP digital signature

Re: Do opaque struct changes break C library ABIs

[Simon McVittie]

On 17/05/13 10:43, Chow Loong Jin wrote:
> On 17/05/2013 13:17, Guillem Jover wrote:
>> I agree dlopen()ing shared libraries in general should not be
>> supported (I'd even go further and say this should be outright 
>> banned, given the pain it causes, and optional library support
>> should always be implemented by loading a plugin properly linked
>> against such library with an ABI under the control of the program
>> loading it).
> 
> But how do you load a plugin without using dlopen()?

Classify each ELF dynamic library as either a "shared library" or a
"plugin", where shared libraries:

* have a proper SONAME (libfoo.so.1 or at least libfoo-1.2.3.so)
* are linked normally (without using -module, if using libtool)
* are in /usr/lib/TUPLE or /usr/lib or a private library directory
* have a compilation symlink (libfoo.so -> libfoo.so.1), if not private

and plugins:

* have a trivial SONAME (libfoo-plugin.so)
* are linked with "libtool --mode=link -module -avoid-version"
  if using libtool
* are always in a private library directory

According to libtool documentation, on some platforms this distinction
is really significant, and "real shared libraries" can't be
dlopen()'d. However, GNU/anything and Windows (and also Mac OS, I
think) are among the platforms where either works, so in practice most
projects don't have any supported platforms where there is a big
technical difference between shared libraries and plugins, and the
line between the two gets blurred.

S


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/51961995.5080...@debian.org

Re: Do opaque struct changes break C library ABIs

[Sune Vuorela]

On 2013-05-17, Simon McVittie  wrote:
> dlopen()'d. However, GNU/anything and Windows (and also Mac OS, I
> think) are among the platforms where either works, so in practice most

You can't link plugins on windows, I'm told.

/Sune


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/slrnkpc80e.j8.nos...@sshway.ssh.pusling.com

Re: Do opaque struct changes break C library ABIs

[Stéphane Glondu]

Le 17/05/2013 13:50, Simon McVittie a écrit :
> According to libtool documentation, on some platforms this distinction
> is really significant, and "real shared libraries" can't be
> dlopen()'d. However, GNU/anything and Windows (and also Mac OS, I
> think) are among the platforms where either works, so in practice most
> projects don't have any supported platforms where there is a big
> technical difference between shared libraries and plugins, and the
> line between the two gets blurred.

Mac OS X (Darwin) does make the distinction, see [1].

[1]
http://stackoverflow.com/questions/2339679/what-are-the-differences-between-so-and-dylib-on-osx

-- 
Stéphane


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/5196247f.30...@debian.org

Re: Do opaque struct changes break C library ABIs

[Stéphane Glondu]

Le 17/05/2013 14:18, Sune Vuorela a écrit :
> On 2013-05-17, Simon McVittie  wrote:
>> dlopen()'d. However, GNU/anything and Windows (and also Mac OS, I
>> think) are among the platforms where either works, so in practice most
> 
> You can't link plugins on windows, I'm told.

Indeed. Some information on the topic is available at:

  http://alain.frisch.fr/flexdll.html

-- 
Stéphane


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/5196262f.4000...@debian.org

Re: Do opaque struct changes break C library ABIs

[Chow Loong Jin]

On 17/05/2013 19:50, Simon McVittie wrote:
> On 17/05/13 10:43, Chow Loong Jin wrote:
>> On 17/05/2013 13:17, Guillem Jover wrote:
>>> I agree dlopen()ing shared libraries in general should not be
>>> supported (I'd even go further and say this should be outright 
>>> banned, given the pain it causes, and optional library support
>>> should always be implemented by loading a plugin properly linked
>>> against such library with an ABI under the control of the program
>>> loading it).
>>
>> But how do you load a plugin without using dlopen()?
> 
> Classify each ELF dynamic library as either a "shared library" or a
> "plugin", where shared libraries:
> 
> * have a proper SONAME (libfoo.so.1 or at least libfoo-1.2.3.so)
> * are linked normally (without using -module, if using libtool)
> * are in /usr/lib/TUPLE or /usr/lib or a private library directory
> * have a compilation symlink (libfoo.so -> libfoo.so.1), if not private
> 
> and plugins:
> 
> * have a trivial SONAME (libfoo-plugin.so)
> * are linked with "libtool --mode=link -module -avoid-version"
>   if using libtool
> * are always in a private library directory
> 
> According to libtool documentation, on some platforms this distinction
> is really significant, and "real shared libraries" can't be
> dlopen()'d. However, GNU/anything and Windows (and also Mac OS, I
> think) are among the platforms where either works, so in practice most
> projects don't have any supported platforms where there is a big
> technical difference between shared libraries and plugins, and the
> line between the two gets blurred.

Okay, so real shared libraries can't be dlopen()'d on some systems, and plugins
still have to be dlopen()'d. That doesn't answer my question, really.

-- 
Kind regards,
Loong Jin



signature.asc
Description: OpenPGP digital signature

Re: Do opaque struct changes break C library ABIs

[Stéphane Glondu]

Le 17/05/2013 15:45, Chow Loong Jin a écrit :
 But how do you load a plugin without using dlopen()?
> [...]
> Okay, so real shared libraries can't be dlopen()'d on some systems, and 
> plugins
> still have to be dlopen()'d. That doesn't answer my question, really.

It kind of does, actually :-) You simply don't load a plugin without
using dlopen().


Cheers,

-- 
Stéphane


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/51964760.5020...@debian.org

Re: Do opaque struct changes break C library ABIs

[Wouter Verhelst]

On 17-05-13 02:35, Russ Allbery wrote:
> I have a C shared library that takes a pointer to an opaque struct as the
> first argument to most of its API calls.  The internal layout of that
> opaque struct is changing (to add new members).  The only way to create
> the opaque state struct is via a call to remctl_new(), which returns a
> pointer to it.  Nothing else about the ABI is changing, and client
> programs cannot see or manipulate the struct members directly (at least
> without poking about in memory, of course).
> 
> Is that an ABI break that requires an SONAME change?

No. The return values of all your functions are still void* (or struct
foo*, where struct foo is not defined for the library user), which has
the exact same signature.

> If you have two versions of the library loaded in memory at the same time,
> call remctl_new() from one version, and then call functions from the other
> version, this will break horribly,

Yes. If you're going to support that, you need symbol versioning. This
is somewhat complicated, and I would recommend against it unless you
really need it.

> since the struct layout has changed and
> will be misinterpreted.  However, all the ABI calls are identical between
> the two versions of the library, so this would require the dynamic linker
> resolve two calls to the same library differently.  Also, if both
> libraries have the same SONAME, I don't see any way that you could load
> two versions into memory at the same time without playing tricks with
> dlopen() that I don't see any reason to support.

If a program links your library statically in but loads plugins (one of
which might also load your library, but dynamically) you have the same
problem. However, doing something like that is a bad idea for multiple
reasons (including this one), so I wouldn't worry about it too much.

> Is that something that one has to guard against, or can one assume that
> the ABI calls from a client of a shared library will be consistent in this
> situation: that all the symbols will resolve to the same version of the
> shared library when both share the same SONAME?
> 
> I'm pretty sure this change is not an ABI break, but the more I thought
> about it, the more I wasn't absolutely certain that I had the right mental
> model here, so I thought I'd ask.

I'm pretty sure you're right.

-- 
This end should point toward the ground if you want to go to space.

If it starts pointing toward space you are having a bad problem and you
will not go to space today.

  -- http://xkcd.com/1133/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/519728ea.6010...@debian.org