RE: Self-initialization of locking/threadid callbacks and auto-detection of features
I like your proposal, but I'd prefer to see an "already initialized" error code returned. Or a flag to the (new?) init api that says "ignore if already set" /r$ -- Principal Security Engineer Akamai Technology Cambridge, MA
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Monday, October 21, 2013, Salz, Rich wrote: > I like your proposal, but I'd prefer to see an "already initialized" error > code returned. Or a flag to the (new?) init api that says "ignore if > already set" > Thanks for your reply! I can add an error, but note that the caller can set then get the callbacks and compare to check whether the caller's callbacks were taken. I could also add a new set of callback setters with ignore-if-set flags. As long as the existing ones behave reliably in the already-set case. In the already-set case I think it may well be best to ignore without failing on the theory that the caller that first set the callbacks must have set sufficiently useful ones anyways... and that where the OS has a good enough default threading library, that's the one that will be used by all DSOs calling OpenSSL in the same process, as otherwise all hell would already be breaking loose anyways! (I can imagine twisted cases where this would not be true, but they seem exceedingly unlikely.) If you want to see the half-baked bits I have (which build on Linux, but which aren't tested) to see what I'm up to, see https://github.com/nicowilliams/openssl, specifically the thread_safety branch. See the XXX comments in rand_lib.c in particular. The outline: add a thread-safe one-time initialization function, built on whatever the OS provides, then use that to make callback init thread-safe. What I need to know: - should i add new targets to ./Configure? for now I modified the linux-elf target, but this feels wrong to me. - what about Windows? I either need to have different targets for pre-vista/2008 or. i have to write a once initialization function for older Windows (which I can and know how to do, it's just more work that, and in particular i couldn't test it, so I'm not inclined to do it). - if so, should ./config automatically pick the new targets where there is appropriate threading support? - how to allocate error codes for "already initialized" errors that you suggest? - should I work to make sure that it's possible to change the default RAND method after it's been set once? The code in rand_lib.c is currently fundamentally thread-unsafe, though it could be accidentally thread-safe if, e.g., ENGINE_finish() doesn't actually tear down state at all. The simplest fix involves setting the default only once, as wih the callbacks, but here I feel that's a shaky idea, that I should allow RAND method changes at any time, in a thread-safe manner -- more work for me, but less surprising. Nico -- (sent from a mobile device with lousy typing options, and no plain text button) (my patches need rebasing to squash and split up, need tests, need finishing, but if you have comments I would love them sooner than later! :)
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On 22 October 2013 06:47, Nico Williams wrote: > On Monday, October 21, 2013, Salz, Rich wrote: > >> I like your proposal, but I'd prefer to see an "already initialized" >> error code returned. Or a flag to the (new?) init api that says "ignore if >> already set" >> > > Thanks for your reply! > > I can add an error, but note that the caller can set then get the > callbacks and compare to check whether the caller's callbacks were taken. > I could also add a new set of callback setters with ignore-if-set flags. > As long as the existing ones behave reliably in the already-set case. > > In the already-set case I think it may well be best to ignore without > failing on the theory that the caller that first set the callbacks must > have set sufficiently useful ones anyways... and that where the OS has a > good enough default threading library, that's the one that will be used by > all DSOs calling OpenSSL in the same process, as otherwise all hell would > already be breaking loose anyways! (I can imagine twisted cases where this > would not be true, but they seem exceedingly unlikely.) > > If you want to see the half-baked bits I have (which build on Linux, but > which aren't tested) to see what I'm up to, see > https://github.com/nicowilliams/openssl, specifically the thread_safety > branch. See the XXX comments in rand_lib.c in particular. The outline: > add a thread-safe one-time initialization function, built on whatever the > OS provides, then use that to make callback init thread-safe. > > What I need to know: > > - should i add new targets to ./Configure? for now I modified the > linux-elf target, but this feels wrong to me. > > - what about Windows? I either need to have different targets for > pre-vista/2008 or. i have to write a once initialization function for older > Windows (which I can and know how to do, it's just more work that, and in > particular i couldn't test it, so I'm not inclined to do it). > > - if so, should ./config automatically pick the new targets where there > is appropriate threading support? > I've been musing about a more autoconf-like approach for some time now (but, for the love of all that is fluffy, not using autoconf itself, which sucks) - it seems this is a good reason to go down that path. Interesting question is: what to do if no appropriate locking mechanism is discovered? > > - how to allocate error codes for "already initialized" errors that you > suggest? > > - should I work to make sure that it's possible to change the default > RAND method after it's been set once? > >The code in rand_lib.c is currently fundamentally thread-unsafe, though > it could be accidentally thread-safe if, e.g., ENGINE_finish() doesn't > actually tear down state at all. The simplest fix involves setting the > default only once, as wih the callbacks, but here I feel that's a shaky > idea, that I should allow RAND method changes at any time, in a thread-safe > manner -- more work for me, but less surprising. > > Nico > -- > > (sent from a mobile device with lousy typing options, and no plain text > button) > (my patches need rebasing to squash and split up, need tests, need > finishing, but if you have comments I would love them sooner than later! :) >
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
> But I could have a target that has a weak dependency on pthreads and is safe when the library is present. Ditto Windows (be unsafe pre-vista/2008, safe in vista/2008 and later, using same OpenSSL DLLs builds). I'd rather add this variation later, after the meat of this work is done, assuming such a variation is desired. Of course, that assumes a degree of ABI compatibility in the OS that may not in fact be available -- a great reason not to go for that...
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Oct 22, 2013 10:28 AM, "Ben Laurie" wrote: > On 22 October 2013 06:47, Nico Williams wrote: >> What I need to know: >> >> - should i add new targets to ./Configure? for now I modified the linux-elf target, but this feels wrong to me. >> >> - what about Windows? I either need to have different targets for pre-vista/2008 or. i have to write a once initialization function for older Windows (which I can and know how to do, it's just more work that, and in particular i couldn't test it, so I'm not inclined to do it). >> >> - if so, should ./config automatically pick the new targets where there is appropriate threading support? > > > I've been musing about a more autoconf-like approach for some time now (but, for the love of all that is fluffy, not using autoconf itself, which sucks) - it seems this is a good reason to go down that path. Well, I'm not signing up for that, not yet anyways! :) Short-term advice will do. I think I'll just add new targets and ./config logic for picking then. The fact that targets are stable-ish is useful, as it allows building whatever targets one can build (or cross-build) on the host. autoconf can't do this, and that's one more reason not to autoconf. > Interesting question is: what to do if no appropriate locking mechanism is discovered? I think for a linux-elf-pthread target the dependency on pthreads should be hard. The old linux-elf target should remain thread-unsafe (I can't make OpenSSL fully thread-safe without a thread library). But I could have a target that has a weak dependency on pthreads and is safe when the library is present. Ditto Windows (be unsafe pre-vista/2008, safe in vista/2008 and later, using same OpenSSL DLLs builds). I'd rather add this variation later, after the meat of this work is done, assuming such a variation is desired. Nico --
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
The simplest fix involves setting the default only once, as wih the callbacks, but here I feel that's a shaky idea, that I should allow RAND method changes at any time, in a thread-safe manner -- more work for me, but less surprising. There is no 'safe' way to do this other than hardwired. Admitted, we have a fairly ugly stack on which to find that out, multiple independently developed lumps of code jammed into the same process, quite a few using dlopen()/dlclose() on other libraries - multiples of them calling the crypto. code. All those lumps of code think they 'own' the crypto. stack - worst case scenario was a dlopen()'d library setting the callbacks, then being unloaded while other parts of the stack were still using crypto. Surprisingly - that still worked on some OS's - but some (like AIX/HPUX) ummap program text immediately on dlclose(). Personally, I'd suggest making it build option to turn off default locking and use whatever the OS provides by default. That'll allow the few corner cases to continue doing whatever wierd things they were doing before, but remove the big risk factor for the vast majority of users. And it is becoming a big risk factor now. It certainly shouldn't be an issue for the OS installed OpenSSL which probably covers most of your users, the only sane choice there is the OS default locking scheme anyway. Peter From: Ben Laurie To: openssl-dev@openssl.org, Date: 23/10/2013 00:33 Subject: Re: Self-initialization of locking/threadid callbacks and auto-detection of features Sent by:owner-openssl-...@openssl.org On 22 October 2013 06:47, Nico Williams wrote: On Monday, October 21, 2013, Salz, Rich wrote: I like your proposal, but I'd prefer to see an "already initialized" error code returned. Or a flag to the (new?) init api that says "ignore if already set" Thanks for your reply! I can add an error, but note that the caller can set then get the callbacks and compare to check whether the caller's callbacks were taken. I could also add a new set of callback setters with ignore-if-set flags. As long as the existing ones behave reliably in the already-set case. In the already-set case I think it may well be best to ignore without failing on the theory that the caller that first set the callbacks must have set sufficiently useful ones anyways... and that where the OS has a good enough default threading library, that's the one that will be used by all DSOs calling OpenSSL in the same process, as otherwise all hell would already be breaking loose anyways! (I can imagine twisted cases where this would not be true, but they seem exceedingly unlikely.) If you want to see the half-baked bits I have (which build on Linux, but which aren't tested) to see what I'm up to, see https://github.com/nicowilliams/openssl, specifically the thread_safety branch. See the XXX comments in rand_lib.c in particular. The outline: add a thread-safe one-time initialization function, built on whatever the OS provides, then use that to make callback init thread-safe. What I need to know: - should i add new targets to ./Configure? for now I modified the linux-elf target, but this feels wrong to me. - what about Windows? I either need to have different targets for pre-vista/2008 or. i have to write a once initialization function for older Windows (which I can and know how to do, it's just more work that, and in particular i couldn't test it, so I'm not inclined to do it). - if so, should ./config automatically pick the new targets where there is appropriate threading support? I've been musing about a more autoconf-like approach for some time now (but, for the love of all that is fluffy, not using autoconf itself, which sucks) - it seems this is a good reason to go down that path. Interesting question is: what to do if no appropriate locking mechanism is discovered? - how to allocate error codes for "already initialized" errors that you suggest? - should I work to make sure that it's possible to change the default RAND method after it's been set once? The code in rand_lib.c is currently fundamentally thread-unsafe, though it could be accidentally thread-safe if, e.g., ENGINE_finish() doesn't actually tear down state at all. The simplest fix involves setting the default only once, as wih the callbacks, but here I feel that's a shaky idea, that I should allow RAND method changes at any time, in a thread-safe manner -- more work for me, but less surprising. Nico -- (sent from a mobile device with lousy typing options, and no plain text button) (my patches need rebasing to squash and split up, need tests, need finishing, but if you have comments I would love them sooner than later! :)
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Wed, Oct 23, 2013 at 08:32:35AM +1000, Peter Waltenberg wrote: > There is no 'safe' way to do this other than hardwired. Admitted, we have a > fairly ugly stack on which to find that out, multiple independently > developed lumps of code jammed into the same process, quite a few using > dlopen()/dlclose() on other libraries - multiples of them calling the > crypto. code. Oh, good point. I think what I'll do is add targets that denote "always use the OS thread library; disallow setting these callbacks", and a corresponding command-line option to ./config. This should be the best option in general because of the possibility of the text for callbacks being unmapped when the provider gets dlclose()ed. Then maybe there's no need to bother with the pthread_once()/ InitOnceExecuteOnce() business. I had assumed, going in, that I needed to preserve existing semantics as much as possible, but because that might still be the case (even if you're right as to what the ideal should be) I will do *both*. (Who knows, maybe there's a program out there that insists on using the gnu pth library and not the OS' native threading library. Or maybe there's no need to support such oddities.) > All those lumps of code think they 'own' the crypto. stack - worst case > scenario was a dlopen()'d library setting the callbacks, then being > unloaded while other parts of the stack were still using crypto. > Surprisingly - that still worked on some OS's - but some (like AIX/HPUX) > ummap program text immediately on dlclose(). Right. That's a question regarding patch contribution: is it OK to "leak" things that can only really be torn down at dlclose() / unload time and which effectively never happens? "git grep '\.fini\>'" finds nothing in the tree (while "git grep '\.init\>' does), so I'm guessing "yes, it's OK". > Personally, I'd suggest making it build option to turn off default locking > and use whatever the OS provides by default. That'll allow the few corner > cases to continue doing whatever wierd things they were doing before, but > remove the big risk factor for the vast majority of users. And it is > becoming a big risk factor now. > > It certainly shouldn't be an issue for the OS installed OpenSSL which > probably covers most of your users, the only sane choice there is the OS > default locking scheme anyway. Agreed. Thanks for your response, Nico -- __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Wed, Oct 23, 2013 at 12:59:53AM -0500, Nico Williams wrote: > On Wed, Oct 23, 2013 at 08:32:35AM +1000, Peter Waltenberg wrote: > > There is no 'safe' way to do this other than hardwired. Admitted, we have a > > fairly ugly stack on which to find that out, multiple independently > > developed lumps of code jammed into the same process, quite a few using > > dlopen()/dlclose() on other libraries - multiples of them calling the > > crypto. code. > > Oh, good point. > > I think what I'll do is add targets that denote "always use the OS > thread library; disallow setting these callbacks", and a corresponding > command-line option to ./config. This should be the best option in > general because of the possibility of the text for callbacks being > unmapped when the provider gets dlclose()ed. > > Then maybe there's no need to bother with the pthread_once()/ > InitOnceExecuteOnce() business. I had assumed, going in, that I needed > to preserve existing semantics as much as possible, but because that > might still be the case (even if you're right as to what the ideal > should be) I will do *both*. (Who knows, maybe there's a program out > there that insists on using the gnu pth library and not the OS' native > threading library. Or maybe there's no need to support such oddities.) You're conserned that you might have 2 libraries in your address space implementing pthreads? That might of course happen, but unless they're using symbol versioning it's going to fail. So I suggest you forget about it and let whoever wants to do that fix things. Kurt __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
No, multiple independently developed libraries in the same process space
calling the same crypto. code was the problem.
Multiple thread models can't work if they call common code, agreed
there :).
The problem we hit early on was that as a library the only way we could
ensure the stack above us was stable was to use the OS default locking
scheme internally, nothing else would work once the complexity started
climbing. We did it as you suggest, hooked the shared library 'init' entry
points and created the locks when the library was loaded.
Peter
From: Kurt Roeckx
To: openssl-dev@openssl.org,
Date: 24/10/2013 06:44
Subject: Re: Self-initialization of locking/threadid callbacks and
auto-detection of features
Sent by:owner-openssl-...@openssl.org
On Wed, Oct 23, 2013 at 12:59:53AM -0500, Nico Williams wrote:
> On Wed, Oct 23, 2013 at 08:32:35AM +1000, Peter Waltenberg wrote:
> > There is no 'safe' way to do this other than hardwired. Admitted, we
have a
> > fairly ugly stack on which to find that out, multiple independently
> > developed lumps of code jammed into the same process, quite a few using
> > dlopen()/dlclose() on other libraries - multiples of them calling the
> > crypto. code.
>
> Oh, good point.
>
> I think what I'll do is add targets that denote "always use the OS
> thread library; disallow setting these callbacks", and a corresponding
> command-line option to ./config. This should be the best option in
> general because of the possibility of the text for callbacks being
> unmapped when the provider gets dlclose()ed.
>
> Then maybe there's no need to bother with the pthread_once()/
> InitOnceExecuteOnce() business. I had assumed, going in, that I needed
> to preserve existing semantics as much as possible, but because that
> might still be the case (even if you're right as to what the ideal
> should be) I will do *both*. (Who knows, maybe there's a program out
> there that insists on using the gnu pth library and not the OS' native
> threading library. Or maybe there's no need to support such oddities.)
You're conserned that you might have 2 libraries in your address
space implementing pthreads? That might of course happen, but
unless they're using symbol versioning it's going to fail. So I
suggest you forget about it and let whoever wants to do that fix
things.
Kurt
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
__
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
I'm making progress slowly (not my main project). I've run into a bit of a problem: the dynlock callback setting cannot be made thread-safe due to the setter API's using three functions to set three related callbacks. Also, I'm not sure that the dynlocks need a default implementation. It seems they don't, at least not as far as *apps* are concerned. Please let me know, though I'm inclined to provide a default implementation anyways, just because it's no big deal. A few notes and questions: - Locks in OpenSSL are really reader-writer locks, but the sample code in crypto/threads/*.c uses mutexes only. How important is it that reader/writer locks be used instead of exclusive locks? - The add_lock stuff should just use OPENSSL_atomic_add() wherever it exists. How would I determine at build-time (in ./Configure and in ./crypto/lock.c) whether OPENSSL_atomic_add is available? - I'll be adding a single setter for dynlock callbacks, and deprecating the old ones. Any objections? - As I get closer to having code that can be tested... I can provide tests of the one-time initialization of things, but it'd be nice to test threaded functionality in general -- is there such a general test? If so, please point me to it. Nico -- __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
I feel like saying few words. One should recognize that by the time multi-threading support was taking shape there was a whole variety of threading implementations and callbacks were the only way to convey the specifics. Nowadays we're pretty much talking only about pthreads and Windows, and one can indeed argue why wouldn't OpenSSL simply default to either of the two when appropriate. While it's more than appropriate on Windows as it is, on pthreads-powered Unices it's not as obvious. Because pthreads can be undesired in some situations. Basically it boils down to question whether or not libcrypto may be linked with libpthread or not. And answer is actually "not desired." Ideally libcrypto should *detect* if *hosting* application is linked with libpthread and only *then* adjust accordingly. Is there way to detect if application is linked with libpthread at run-time? There is DSO_global_lookup. As for pthread_once [and Windows counterpart]. I'd argue that it would be more appropriate to initialize the default callbacks through initialization segment (facility used for "pre-main" execution of "constructors"), which is guaranteed to be MT-safe. Such framework (as well as "destructors") would be useful in more places. As for config-time detection. My major objection is following. Software is called so for a reason, and config-time detection is effectively against its nature. Because it "hardwires" it to specific set of interfaces. Wouldn't it be more natural for *soft*ware to detect feature at *run-time* and adapt? This is actually what's is implemented in several places in OpenSSL. I'm making progress slowly (not my main project). I've run into a bit of a problem: the dynlock callback setting cannot be made thread-safe due to the setter API's using three functions to set three related callbacks. Also, I'm not sure that the dynlocks need a default implementation. It seems they don't, at least not as far as *apps* are concerned. Please let me know, though I'm inclined to provide a default implementation anyways, just because it's no big deal. A few notes and questions: - Locks in OpenSSL are really reader-writer locks, but the sample code in crypto/threads/*.c uses mutexes only. How important is it that reader/writer locks be used instead of exclusive locks? For historical reasons mentioned above mutexes are common denominator. Whether or not one should switch to more sophisticated primitives is question about whether or not do we want to support legacy multi-threading systems. The answer is probably yes. Of course one can add more callbacks, see which one are set and opt for more refined methods if available. But - The add_lock stuff should just use OPENSSL_atomic_add() wherever it exists. How would I determine at build-time (in ./Configure and in ./crypto/lock.c) whether OPENSSL_atomic_add is available? #ifdef OPENSSL_CPUID_OBJ Well, DSO_global_lookup is more consistent with above:-) While we are at it, it's appropriate to add atomic pointer swap that can be used to maintain linked lists without mutex-ing. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Tue, Oct 29, 2013 at 09:58:25PM +0100, Andy Polyakov wrote: > I feel like saying few words. One should recognize that by the time > multi-threading support was taking shape there was a whole variety > of threading implementations and callbacks were the only way to > convey the specifics. Nowadays we're pretty much talking only about Certainly. I'm not saying "omg, that OpenSSL is insanez!!". I should repeat my problem case: we have libraries that depend on OpenSSL libraries, and it should be possible to for our libraries to work in threaded programs. But right now a Java program that uses JGSS with the JNI shim to the C GSS-API libraries using Heimdal's libgss crashes in RAND_bytes(). Heimdal can't provide locking callbacks (how could it? it might step on some other caller's toes in the same process). The app can't either (since in this case it's not using OpenSSL directly at all). > pthreads and Windows, and one can indeed argue why wouldn't OpenSSL > simply default to either of the two when appropriate. While it's > more than appropriate on Windows as it is, on pthreads-powered > Unices it's not as obvious. Because pthreads can be undesired in > some situations. Basically it boils down to question whether or not > libcrypto may be linked with libpthread or not. And answer is > actually "not desired." Ideally libcrypto should *detect* if More details would be nice. What follows is speculation of mine as to what you might have in mind (or even if you don't, it might be relevant). To summarize the below: OpenSSL .so's should always be linked with -lpthread on OSes where there's a standard libpthread that threaded apps are expected to use; OpenSSL static link archives should assume no threading libraries and rely on the callers to provide the threading callbacks. Distros that focus on static linking should ship OpenSSL static link archives only and shuld ensure that dependents of OpenSSL initialize it correctly when used in threaded programs. Linux still has the multiple process models problem that Solaris <9 used to have: - statically linked with libc but not -lpthread - statically linked with libc and -lpthread - dynamically linked with libc but not -lpthread - dynamically linked with libc and -lpthread - mixed: not originally linked with -lpthread but a dlopen()ed object brought it in I dunno about Linux, but IIUC loading libpthread at run-time in the third case works fine. It also works fine in S8 and S9. It's loading libpthread at run-time in the first case that leads to fireworks. The mixed case is a very difficult one to handle correctly in any library. Merely using dlsym() on RTLD_DEFAULT is not good enough: pthreads symbols might appear later on. Typically the mixed case results when an app uses the name service switch without nscd, or PAM, and one of those pluggable things loads libcrypto. If the app also uses libcrypto then... who knows. (It's clearly insane to support a static libc and have an RTLD. And yet Solaris used to support that... But then, it stopped supporting that for a reason: it was insanely expensive in terms of engineering costs.) I'm not sure that OpenSSL should support such madness in any way other than at ./Configure time: the builder person picks. I.e., this is the distro builder's problem. Some distros are very focused on static linking. Others are not. Static link archives don't record dependencies, so we can't speak of building libcrypto.a with -lpthread. So in the static link build option OpenSSL needs to know whether to assume pthreads or not, and only the builder can tell OpenSSL. In the shared object build of OpenSSL it's just easier to link with -lpthread if the target has pthreads. E.g., a distro that is mostly-dynamically-linked should just ship OpenSSL libs linked with -lpthread, while a distro that is mostly-statically-linked should ship OpenSSL .a's built to not use pthreads. A distro with both should *probably* build OpenSSL .a's w/o pthread and .so's with. A distro might want to require that pthreads is *always* available, even with linking statically. Or perhaps you were thinking of apps that use GNU Pth, but those should either build their own private OpenSSL libs or they should use static link OpenSSL archives. I don't think OpenSSL should bend over backwards to support third-party alternate threading libraries where the OS has a single supported standard. If an OS has multiple supported alternatives then OpenSSL should pick one at run-time, but this is very difficult (see below). > *hosting* application is linked with libpthread and only *then* > adjust accordingly. Is there way to detect if application is linked > with libpthread at run-time? There is DSO_global_lookup. See above. Using weak symbols might work, but it might be very OS-dependent; I'm not prepared to research that for every target, or even more than one or two targets. Using dlsym() seems like asking for troub
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Oct 29 21:58, Andy Polyakov wrote: > I feel like saying few words. One should recognize that by the time > multi-threading support was taking shape there was a whole variety > of threading implementations and callbacks were the only way to > convey the specifics. Nowadays we're pretty much talking only about > pthreads and Windows, and one can indeed argue why wouldn't OpenSSL > simply default to either of the two when appropriate. While it's > more than appropriate on Windows as it is, on pthreads-powered > Unices it's not as obvious. > [...] > Ideally libcrypto should *detect* if > *hosting* application is linked with libpthread and only *then* > adjust accordingly. Is there way to detect if application is linked > with libpthread at run-time? There is DSO_global_lookup. Please, before any change is made in terms of threading, let me point out that Cygwin is NOT Windows, even if it runs on Windows. Cygwin provides its own pthreads implementation which is always available, even without explicit linking against -lpthread. So, for Cygwin, please use pthreads, not Windows threading, otherwise applications linked against libcrypto might be broken in subtil ways. Thanks for considering, Corinna -- Corinna Vinschen Cygwin Maintainer Red Hat pgpKeAfpitjTD.pgp Description: PGP signature
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Wed, Oct 30, 2013 at 11:15:27AM +0100, Corinna Vinschen wrote: > Please, before any change is made in terms of threading, let me point > out that Cygwin is NOT Windows, even if it runs on Windows. Cygwin > provides its own pthreads implementation which is always available, > even without explicit linking against -lpthread. > > So, for Cygwin, please use pthreads, not Windows threading, otherwise > applications linked against libcrypto might be broken in subtil ways. Of course. But surely OpenSSL needs to be built separately (and differently) for native Windows and Cygwin. I.e., an OpenSSL DLL built against the Windows CRT can't be used from a Cygwin app. __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Oct 30 11:39, Nico Williams wrote: > On Wed, Oct 30, 2013 at 11:15:27AM +0100, Corinna Vinschen wrote: > > Please, before any change is made in terms of threading, let me point > > out that Cygwin is NOT Windows, even if it runs on Windows. Cygwin > > provides its own pthreads implementation which is always available, > > even without explicit linking against -lpthread. > > > > So, for Cygwin, please use pthreads, not Windows threading, otherwise > > applications linked against libcrypto might be broken in subtil ways. > > Of course. But surely OpenSSL needs to be built separately (and > differently) for native Windows and Cygwin. I.e., an OpenSSL DLL built > against the Windows CRT can't be used from a Cygwin app. Right. I'm just trying to raise awareness before Cygwin gets sorted into the "runs on Windows so use Windows functions" drawer, accidentally, as happened too often in the past in various projects. Corinna -- Corinna Vinschen Cygwin Maintainer Red Hat pgpV35LME50WJ.pgp Description: PGP signature
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Wed, Oct 30, 2013 at 06:12:08PM +0100, Corinna Vinschen wrote: > Right. I'm just trying to raise awareness before Cygwin gets sorted > into the "runs on Windows so use Windows functions" drawer, accidentally, > as happened too often in the past in various projects. I won't. I'll basically be adding new targets for every existing target that has a standard (POSIX or Win32) threading library. I probably won't change what target ./config picks -- I think core OpenSSL developers must make that decision. Nico -- __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Tue, Oct 29, 2013 at 09:58:25PM +0100, Andy Polyakov wrote: > Another thing: run-time selection from among multiple pthreads implementations can only safely be managed by the RTLD, and dependencies on pthreads must still be declared via the linker-editor. Libraries should be able to refer to pthread_* symbols and still use the run-time-selected pthread implementation *even* if the library was linked with -lpthread (and therefore a specific libpthread.so compilation symlink). If an OS delivers two (or more) pthreads implementations then: - if the ABIs of those pthreads implementations all match then the RTLD should be able to substitute any of them regardless of which libpthread any one object in the process was linked with; (e.g., via LD_PRELOAD, different library search paths, ..., but all the libpthreads would have to have the same SONAMEs and symbol version numbers) - but if their ABIs are not compatible then no run-time substitution is safe, no matter what the mechanism. (unless compatibility shims are involved, but then we *really* must leave it all to the RTLD and we can't be using dlsym() or weak symbols) I don't think libraries should be failing to declare their dependencies and then use dlsym(RTLD_DEFAULT) and dlopen()+dlsym() as a way to run-time link-edit-load their undeclared dependencies. Doing so is fundamentally racy and so would have to be done in .init sections, and results (which pthread gets used) would be unpredictable (since that would depend on object load and .init firing order). Plus we'd lose observability via ldd(1). I can't see any way to justify this without having examples of OSes where this is the documented way to handle dynamic linking against libpthread. But maybe I'm missing something? Nico -- __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
It's getting closer, see: https://github.com/nicowilliams/openssl/compare/thread_safety I spoke to a Linux expert and was told that the correct thing to do in the shared build of OpenSSL is to always link with -lpthread. I assert that this is also the correct thing to do on Solaris. It's been so for a long time now. I.e., OpenSSL should default to using native threads on all POSIX-like and Win32/64 systems (others will have to contribute similar support for other environments). My patches nonetheless will allow apps to provide alternate threading callbacks, just as today, except that the callbacks now may not be changed once set, and they are set to native implementations where possible if they are needed before they are set. Nico -- __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager majord...@openssl.org
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
Not to stand in the way of progress at all, but just to note we cross-compile OpenSSL libraries for an embedded linux target that is still using libc_r, not libpthread. That will not change anytime soon for us, at least on legacy systems. Besides that, it seems much of this discussion is about native builds and config-time detection of threading support. What about cross-compiling? Kevin On Wed, Oct 30, 2013 at 7:38 PM, Nico Williams wrote: > > It's getting closer, see: > > https://github.com/nicowilliams/openssl/compare/thread_safety > > I spoke to a Linux expert and was told that the correct thing to do in > the shared build of OpenSSL is to always link with -lpthread. I assert > that this is also the correct thing to do on Solaris. It's been so for > a long time now. I.e., OpenSSL should default to using native threads > on all POSIX-like and Win32/64 systems (others will have to contribute > similar support for other environments). > > My patches nonetheless will allow apps to provide alternate threading > callbacks, just as today, except that the callbacks now may not be > changed once set, and they are set to native implementations where > possible if they are needed before they are set. > > Nico > -- > __ > OpenSSL Project http://www.openssl.org > Development Mailing List openssl-dev@openssl.org > Automated List Manager majord...@openssl.org >
Re: Self-initialization of locking/threadid callbacks and auto-detection of features
On Oct 31, 2013 8:19 AM, "Kevin Fowler" wrote: > > Not to stand in the way of progress at all, but just to note we cross-compile OpenSSL libraries for an embedded linux target that is still using libc_r, not libpthread. That will not change anytime soon for us, at least on legacy systems. > > Besides that, it seems much of this discussion is about native builds and config-time detection of threading support. What about cross-compiling? I will be introducing new targets. For cross-compilation you must tell ./Configure the target name and there's no auto-detection. If you want auto-detection you run ./config on the target host. (At least that's my understanding of how OpenSSL build configuration works.)
