We hit the same problem with IBM code.
You simply can't resolve this "externally" when you have multiple shared
libraries that were built independently stuffed into the same process - and
in our case most of those libs were originally applications in their own
right (so think they are "god" anyway).
The Linux distro's are getting to be almost as bad^H^H^H complex as IBM's
code now so I'm not surprised this is being hit by others.
I "fixed" it in our code by setting the callbacks internally and ignoring
attempts to replace them - closed ecosystem, more or less, so I could use a
sledgehammer.
A couple of suggestions, allow the threading callbacks to be set up once
only and once only - that way whoever is "main()" gets first crack at it
anyway.
Un-registering the callbacks is a problem still, if it's possible to detect
that no threads are running (not a problem I had to solve so I don't know
it's possible), refusing to un-register callbacks while threads are active
should work mostly.
Alternatively, add a build flag -DUSE_NATIVE_THREADS and do what I did if
that's set - build in the native threading support in at runtime. I can
provide code/macro's which will do this for most mainstream OS's if it'll
help.
That's probably a good fix for the Linux distro's/BSD's and probably most
other OS vendors. The only OS I know for sure could have problems with
implementing OpenSSL that way would be HP/UX - but I suspect they have to
build OpenSSL threaded anyway to avoid crashes.
This isn't *my* problem, I resolved it for our usage, but I certainly
understand where Michael is coming from, and I can also confirm that it's
not a problem which has an external fix when you don't have direct control
over the entire software stack.
Peter
From: "Michael Sweet via RT" <r...@openssl.org>
To:
Cc: openssl-dev@openssl.org
Date: 06/25/2010 01:23 PM
Subject: Re: [openssl.org #2293] OpenSSL dependence on external threading
functions is a critical design flaw
Sent by: owner-openssl-...@openssl.org
On Jun 24, 2010, at 4:58 PM, Ger Hobbelt via RT wrote:
> The only critical bit here is opinion replacing analysis.
I won't get into a personal attack.
> As you said so yourself, and I quote: "When both an application and a
> library ([...]) both initialize and register their own threading
functions,
> bad things happen and the application can crash.")
> Indeed. Spot on, I'd say.
>
> That is /exactly/ why OpenSSL places the ability, and hence also the
> responsibility [which is the evil siamese twin of 'ability'], to
_provide_
> an singular set of locking operations to the OpenSSL library itself for
its
> use.
With all due respect, OpenSSL is being used as a core/standard library on
Linux, the BSD's, and to a lesser extent on the commercial UNIX's. Every
standard library has built-in support for multi-threading, and depending on
the library that support may come via a series of platform-specific
abstraction modules. An application might need to provide its own
platform-specific modules, but never for the standard libraries it uses.
> ...
> Ranting aside, the problem, if /any/, is not OpenSSL this time, but it's
> /environment/, either other libraries using it, where the designers may
have
> missed this spot (read: they don't 'propagate' threading/locking
primitives
> specification to the next upper design block level) or the application
> [programmer] itself, who'd rather not have the resposibility, so he can
> spare his remaining synapses for other use. In doing so, they assume a
lot,
> including OpenSSL being there for them specifically, while OpenSSL is a
> library meant (and designed) for a plethora of environments, including
quite
> a few that definitely do sport requirements agonizingly perpendicular to
> yours.
Having struggled over the years to get OpenSSL (and CUPS, for that matter)
running on diverse platforms, and having done a LOT of bare metal
development early in my career, I can appreciate wanting to keep any
system-level interfaces out of a toolkit and keeping the code as simple as
possible.
That said, OpenSSL already exposes POSIX concepts like file descriptors and
filenames, even though many platforms do not use file descriptors or
filenames natively at all. At some level you need to decide: do you want
OpenSSL to be a low-level toolkit that can't be used by applications
directly, or do you want it to be a standard library where basic, core
functionality is implemented by the library and not by the calling
application?
> I have (had to) port several other libraries to embedded/non-UNIX
> environments (most of those libs originating with the GNU/BSD/Linux
> collective and 'of course' carrying all sorts of assumptions that are
truish
> within that originating environment) and quite a few of them had
threading /
> locking demands. The only one that was rather plug-and-play was OpenSSL;
I've never found the OpenSSL build system to be "plug-and-play"...
> I have not had the pleasure to port libcups yet, but I can imagine they
> succumbed to those 'we're on UNIX, guys!' assumptions (for good reason, I
(FWIW, the "they" for CUPS is mostly "me"...)
Actually, no. There are certain assumptions that I've made over the years
(mainly that there will be a BSD-like sockets interface and 32-bit or
larger ints), but overall the code has proven very portable and I have
carefully abstracted problematic interfaces (like encryption, file IO,
memory allocation, threading, and mutexes) so that porting is often just a
matter of providing a new implementation of those interfaces specific to
the platform.
Of course, by doing so I've made libcups marginally larger and harder to
maintain. However, all of the applications that use libcups know that they
don't need to bother with OS-specific stuff unless they want to, and I'd
hazard a guess that the maintenance is far less involved than, say, keeping
a lot of hand-optimized assembly for platform X working.
> ...
> Be glad OpenSSL does provide these callbacks, the way it does. Yes, it's
a
> couple of lines more code to write then when 'they' (for any amount of
> 'they') had provided you with a ready-to-go implementation custom
tailored
> for your rig, but that 'lack' can be quickly alleviated by looking for a
bit
> of sample code, some judicious copy&paste&expand-on-that and you're good
to
> go. Besides, you get to have the wonderful sense of accomplishment that
day.
> Which is a valuable benefit, considering the usual paycheck.
I find no benefit in re-implementing the same code multiple times,
particularly when my implementation will likely conflict with the next
developer's implementation (even if it is the same code!)
I think it is great that you have abstracted all of the locking interfaces
within OpenSSL. I took a similar approach in CUPS and other libraries I
have done. However, I also provided implementations for the popular
platforms (that I was willing to support directly) in those projects
themselves to provide the greatest benefits - people can still port my
projects to new platforms but they only have to implement/instantiate the
platform-specific code *once* for all of the applications that use that
project instead of N times. Moreover, by putting that platform code in the
project I avoid the side-effects we see with OpenSSL usage on Linux, usage
which is predictable when a library becomes a standard part of an OS and is
used by multiple, separately-maintained bits of code.
> ...
> On Thu, Jun 24, 2010 at 10:19 PM, Michael Sweet via RT
<r...@openssl.org>wrote:
>
>> An emerging issue with CUPS and other applications that use it has
>> identified a critical design flaw in how OpenSSL supports
multithreading.
>>
>> Specifically, a multi-threaded application or library that uses OpenSSL
>> must provide its own threading functions for OpenSSL to use. When both
an
>> application and a library (say, Firefox and libcups) both initialize and
>> register their own threading functions, bad things happen and the
>> application can crash.
>>
>> Pushing the responsibility of making OpenSSL thread-safe on the
application
>> is a bad design choice and needs to be fixed ASAP to allow
cross-platform,
>> multi-threaded applications to be developed safely.
>>
>> ________________________________________________________________________
>> Michael Sweet, Senior Printing System Engineer, PWG Chair
>>
>>
>>
>>
>> ______________________________________________________________________
>> OpenSSL Project http://www.openssl.org
>> Development Mailing List openssl-dev@openssl.org
>> Automated List Manager majord...@openssl.org
>>
>
>
>
> --
> Met vriendelijke groeten / Best regards,
>
> Ger Hobbelt
>
> --------------------------------------------------
> web: http://www.hobbelt.com/
> http://www.hebbut.net/
> mail: g...@hobbelt.com
> mobile: +31-6-11 120 978
> --------------------------------------------------
>
> The only critical bit here is opinion replacing analysis.
>
> As you said so yourself, and I quote: "When both an application and a
library ([...]) both initialize and register their own threading functions,
bad things happen and the application can crash.")
> Indeed. Spot on, I'd say.
>
> That is /exactly/ why OpenSSL places the ability, and hence also the
responsibility [which is the evil siamese twin of 'ability'], to _provide_
an singular set of locking operations to the OpenSSL library itself for its
use.
> These are
> CRYPTO_set_lock_callback
> CRYPTO_add_locking_callback
> CRYPTO_set_dynlock_create_callback
> CRYPTO_set_dynlock_lock_callback
> CRYPTO_set_dynlock_destroy_callback
> and you must call those at library setup time, say, around the same time
you invoke ERR_load_crypto_strings().
>
>
> [The next bit possibly is not going to adhere to soothingly gentle
didactic doctrine, but I'm old enough to still appreciate the sentiment of
Proverbs 13:24 at times.]
>
> To put not to fine a point on it, I wonder how I could have missed such a
critical issue for the last 11 years, while I've quite successfully used
OpenSSL in (and ported the library itself to!) several environments,
including embedded environments (and about 'embedded': I consider only
those environments 'embedded' which are not simply a Linux/UNIX
port/emulation - those are for tenderfoots. ;-) <evil smirk> In a truly
embedded environment, you /don't/ have the luxury of stdin/stdout, let
alone stderr, to name but one. The only standard thing there is you ain't
got POSIX nor any ISO-ish 'standard' run-time library, to name another.
Either you get it right the first time so your serial connection or network
connection actually works, or you're down to hack a kind of POST code with
the few LEDs you've got (if any), or you're hopefully going to be that
lucky s.o.b. with a cuddly budget who's debugging using some fancy JTAG
hardware - or an ICE if you get to be _really_ posh.
>
>
> Ranting aside, the problem, if /any/, is not OpenSSL this time, but
it's /environment/, either other libraries using it, where the designers
may have missed this spot (read: they don't 'propagate' threading/locking
primitives specification to the next upper design block level) or the
application [programmer] itself, who'd rather not have the resposibility,
so he can spare his remaining synapses for other use. In doing so, they
assume a lot, including OpenSSL being there for them specifically, while
OpenSSL is a library meant (and designed) for a plethora of environments,
including quite a few that definitely do sport requirements agonizingly
perpendicular to yours.
>
> I have (had to) port several other libraries to embedded/non-UNIX
environments (most of those libs originating with the GNU/BSD/Linux
collective and 'of course' carrying all sorts of assumptions that are
truish within that originating environment) and quite a few of them had
threading / locking demands. The only one that was rather plug-and-play was
OpenSSL; those others all had threading/locking support 'built-in' to make
app devs happy and I had to forcibly kick that code out of there (and
substitute it, of course) before I'd even have a chance to get them to work
in my environments - given the permeativeness of those elemental bits
throughout the code, I can only say doing that sort of 'portability prep
work' can do wonders for my temper.
>
> I have not had the pleasure to port libcups yet, but I can imagine they
succumbed to those 'we're on UNIX, guys!' assumptions (for good reason, I
think) and you run into those assumptions there, got fazed, fizzed, and the
next grep around the corner landing in OpenSSL County delivered a Blame
Pointer right when you needed one. No hard feelings, but this is not a bug,
but the difference between a multi-platform library and library/application
code which assumes a certain predefined platform set from the get-go.
>
> Be glad OpenSSL does provide these callbacks, the way it does. Yes, it's
a couple of lines more code to write then when 'they' (for any amount of
'they') had provided you with a ready-to-go implementation custom tailored
for your rig, but that 'lack' can be quickly alleviated by looking for a
bit of sample code, some judicious copy&paste&expand-on-that and you're
good to go. Besides, you get to have the wonderful sense of accomplishment
that day. Which is a valuable benefit, considering the usual paycheck.
>
>
>
>
> Note/By The Way:
> When you see the call CRYPTO_thread_setup() invoked
somewhere, /anywhere/, then you've got your hands on a piece of code which
uses severely antiquated support code from OpenSSL (the fossil is still
buried in crypto/threads/th-lock.c. When you feel like it, compare that
code with the list of methods specified at top and arrive to your own
conclusion there. Before anyone feels like piping up right after that
happens: do another grep to verify whether that code is 'visible' outside
OpenSSL (hint: prototypes don't feature in any header file; for good
reason))
>
>
>
> On Thu, Jun 24, 2010 at 10:19 PM, Michael Sweet via RT <r...@openssl.org>
wrote:
> An emerging issue with CUPS and other applications that use it has
identified a critical design flaw in how OpenSSL supports multithreading.
>
> Specifically, a multi-threaded application or library that uses OpenSSL
must provide its own threading functions for OpenSSL to use. When both an
application and a library (say, Firefox and libcups) both initialize and
register their own threading functions, bad things happen and the
application can crash.
>
> Pushing the responsibility of making OpenSSL thread-safe on the
application is a bad design choice and needs to be fixed ASAP to allow
cross-platform, multi-threaded applications to be developed safely.
>
> ________________________________________________________________________
> Michael Sweet, Senior Printing System Engineer, PWG Chair
>
>
>
>
> ______________________________________________________________________
> OpenSSL Project http://www.openssl.org
> Development Mailing List openssl-dev@openssl.org
> Automated List Manager majord...@openssl.org
>
>
>
> --
> Met vriendelijke groeten / Best regards,
>
> Ger Hobbelt
>
> --------------------------------------------------
> web: http://www.hobbelt.com/
> http://www.hebbut.net/
> mail: g...@hobbelt.com
> mobile: +31-6-11 120 978
> --------------------------------------------------
>
________________________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair
On Jun 24, 2010, at 4:58 PM, Ger Hobbelt via RT wrote:
The only critical bit here is opinion replacing analysis.
I won't get into a personal attack.
As you said so yourself, and I quote: "When both an application and a
library ([...]) both initialize and register their own threading
functions,
bad things happen and the application can crash.")
Indeed. Spot on, I'd say.
That is /exactly/ why OpenSSL places the ability, and hence also the
responsibility [which is the evil siamese twin of 'ability'], to
_provide_
an singular set of locking operations to the OpenSSL library itself
for its
use.
With all due respect, OpenSSL is being used as a core/standard library on
Linux, the BSD's, and to a lesser extent on the commercial UNIX's. Every
standard library has built-in support for multi-threading, and depending on
the library that support may come via a series of platform-specific
abstraction modules. An application might need to provide its own
platform-specific modules, but never for the standard libraries it uses.
...
Ranting aside, the problem, if /any/, is not OpenSSL this time, but
it's
/environment/, either other libraries using it, where the designers
may have
missed this spot (read: they don't 'propagate' threading/locking
primitives
specification to the next upper design block level) or the
application
[programmer] itself, who'd rather not have the resposibility, so he
can
spare his remaining synapses for other use. In doing so, they assume
a lot,
including OpenSSL being there for them specifically, while OpenSSL is
a
library meant (and designed) for a plethora of environments,
including quite
a few that definitely do sport requirements agonizingly perpendicular
to
yours.
Having struggled over the years to get OpenSSL (and CUPS, for that matter)
running on diverse platforms, and having done a LOT of bare metal
development early in my career, I can appreciate wanting to keep any
system-level interfaces out of a toolkit and keeping the code as simple as
possible.
That said, OpenSSL already exposes POSIX concepts like file descriptors and
filenames, even though many platforms do not use file descriptors or
filenames natively at all. At some level you need to decide: do you want
OpenSSL to be a low-level toolkit that can't be used by applications
directly, or do you want it to be a standard library where basic, core
functionality is implemented by the library and not by the calling
application?
I have (had to) port several other libraries to embedded/non-UNIX
environments (most of those libs originating with the GNU/BSD/Linux
collective and 'of course' carrying all sorts of assumptions that are
truish
within that originating environment) and quite a few of them had
threading /
locking demands. The only one that was rather plug-and-play was
OpenSSL;
I've never found the OpenSSL build system to be "plug-and-play"...
I have not had the pleasure to port libcups yet, but I can imagine
they
succumbed to those 'we're on UNIX, guys!' assumptions (for good
reason, I
(FWIW, the "they" for CUPS is mostly "me"...)
Actually, no. There are certain assumptions that I've made over the years
(mainly that there will be a BSD-like sockets interface and 32-bit or
larger ints), but overall the code has proven very portable and I have
carefully abstracted problematic interfaces (like encryption, file IO,
memory allocation, threading, and mutexes) so that porting is often just a
matter of providing a new implementation of those interfaces specific to
the platform.
Of course, by doing so I've made libcups marginally larger and harder to
maintain. However, all of the applications that use libcups know that they
don't need to bother with OS-specific stuff unless they want to, and I'd
hazard a guess that the maintenance is far less involved than, say, keeping
a lot of hand-optimized assembly for platform X working.
...
Be glad OpenSSL does provide these callbacks, the way it does. Yes,
it's a
couple of lines more code to write then when 'they' (for any amount
of
'they') had provided you with a ready-to-go implementation custom
tailored
for your rig, but that 'lack' can be quickly alleviated by looking
for a bit
of sample code, some judicious copy&paste&expand-on-that and you're
good to
go. Besides, you get to have the wonderful sense of accomplishment
that day.
Which is a valuable benefit, considering the usual paycheck.
I find no benefit in re-implementing the same code multiple times,
particularly when my implementation will likely conflict with the next
developer's implementation (even if it is the same code!)
I think it is great that you have abstracted all of the locking interfaces
within OpenSSL. I took a similar approach in CUPS and other libraries I
have done. However, I also provided implementations for the popular
platforms (that I was willing to support directly) in those projects
themselves to provide the greatest benefits - people can still port my
projects to new platforms but they only have to implement/instantiate the
platform-specific code *once* for all of the applications that use that
project instead of N times. Moreover, by putting that platform code in the
project I avoid the side-effects we see with OpenSSL usage on Linux, usage
which is predictable when a library becomes a standard part of an OS and is
used by multiple, separately-maintained bits of code.
...
On Thu, Jun 24, 2010 at 10:19 PM, Michael Sweet via RT <
r...@openssl.org>wrote:
An emerging issue with CUPS and other applications that use it
has
identified a critical design flaw in how OpenSSL supports
multithreading.
Specifically, a multi-threaded application or library that uses
OpenSSL
must provide its own threading functions for OpenSSL to use.
When both an
application and a library (say, Firefox and libcups) both
initialize and
register their own threading functions, bad things happen and
the
application can crash.
Pushing the responsibility of making OpenSSL thread-safe on the
application
is a bad design choice and needs to be fixed ASAP to allow
cross-platform,
multi-threaded applications to be developed safely.
________________________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair
______________________________________________________________________
OpenSSL Project
http://www.openssl.org
Development Mailing List
openssl-dev@openssl.org
Automated List Manager
majord...@openssl.org
--
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
http://www.hebbut.net/
mail: g...@hobbelt.com
mobile: +31-6-11 120 978
--------------------------------------------------
The only critical bit here is opinion replacing analysis.
As you said so yourself, and I quote: "When both an application and a
library ([...]) both initialize and register their own threading
functions, bad things happen and the application can crash.")
Indeed. Spot on, I'd say.
That is /exactly/ why OpenSSL places the ability, and hence also the
responsibility [which is the evil siamese twin of 'ability'], to
_provide_ an singular set of locking operations to the OpenSSL
library itself for its use.
These are
CRYPTO_set_lock_callback
CRYPTO_add_locking_callback
CRYPTO_set_dynlock_create_callback
CRYPTO_set_dynlock_lock_callback
CRYPTO_set_dynlock_destroy_callback
and you must call those at library setup time, say, around the same
time you invoke ERR_load_crypto_strings().
[The next bit possibly is not going to adhere to soothingly gentle
didactic doctrine, but I'm old enough to still appreciate the
sentiment of Proverbs 13:24 at times.]
To put not to fine a point on it, I wonder how I could have missed
such a critical issue for the last 11 years, while I've quite
successfully used OpenSSL in (and ported the library itself to!)
several environments, including embedded environments (and about
'embedded': I consider only those environments 'embedded' which are
not simply a Linux/UNIX port/emulation - those are for
tenderfoots. ;-) <evil smirk> In a truly embedded environment,
you /don't/ have the luxury of stdin/stdout, let alone stderr, to
name but one. The only standard thing there is you ain't got POSIX
nor any ISO-ish 'standard' run-time library, to name another. Either
you get it right the first time so your serial connection or network
connection actually works, or you're down to hack a kind of POST code
with the few LEDs you've got (if any), or you're hopefully going to
be that lucky s.o.b. with a cuddly budget who's debugging using some
fancy JTAG hardware - or an ICE if you get to be _really_ posh.
Ranting aside, the problem, if /any/, is not OpenSSL this time, but
it's /environment/, either other libraries using it, where the
designers may have missed this spot (read: they don't 'propagate'
threading/locking primitives specification to the next upper design
block level) or the application [programmer] itself, who'd rather not
have the resposibility, so he can spare his remaining synapses for
other use. In doing so, they assume a lot, including OpenSSL being
there for them specifically, while OpenSSL is a library meant (and
designed) for a plethora of environments, including quite a few that
definitely do sport requirements agonizingly perpendicular to yours.
I have (had to) port several other libraries to embedded/non-UNIX
environments (most of those libs originating with the GNU/BSD/Linux
collective and 'of course' carrying all sorts of assumptions that are
truish within that originating environment) and quite a few of them
had threading / locking demands. The only one that was rather
plug-and-play was OpenSSL; those others all had threading/locking
support 'built-in' to make app devs happy and I had to forcibly kick
that code out of there (and substitute it, of course) before I'd even
have a chance to get them to work in my environments - given the
permeativeness of those elemental bits throughout the code, I can
only say doing that sort of 'portability prep work' can do wonders
for my temper.
I have not had the pleasure to port libcups yet, but I can imagine
they succumbed to those 'we're on UNIX, guys!' assumptions (for good
reason, I think) and you run into those assumptions there, got fazed,
fizzed, and the next grep around the corner landing in OpenSSL County
delivered a Blame Pointer right when you needed one. No hard
feelings, but this is not a bug, but the difference between a
multi-platform library and library/application code which assumes a
certain predefined platform set from the get-go.
Be glad OpenSSL does provide these callbacks, the way it does. Yes,
it's a couple of lines more code to write then when 'they' (for any
amount of 'they') had provided you with a ready-to-go implementation
custom tailored for your rig, but that 'lack' can be quickly
alleviated by looking for a bit of sample code, some judicious
copy&paste&expand-on-that and you're good to go. Besides, you get to
have the wonderful sense of accomplishment that day. Which is a
valuable benefit, considering the usual paycheck.
Note/By The Way:
When you see the call CRYPTO_thread_setup() invoked
somewhere, /anywhere/, then you've got your hands on a piece of code
which uses severely antiquated support code from OpenSSL (the fossil
is still buried in crypto/threads/th-lock.c. When you feel like it,
compare that code with the list of methods specified at top and
arrive to your own conclusion there. Before anyone feels like piping
up right after that happens: do another grep to verify whether that
code is 'visible' outside OpenSSL (hint: prototypes don't feature in
any header file; for good reason))
On Thu, Jun 24, 2010 at 10:19 PM, Michael Sweet via RT <
r...@openssl.org> wrote:
An emerging issue with CUPS and other applications that use it has
identified a critical design flaw in how OpenSSL supports
multithreading.
Specifically, a multi-threaded application or library that uses
OpenSSL must provide its own threading functions for OpenSSL to
use. When both an application and a library (say, Firefox and
libcups) both initialize and register their own threading
functions, bad things happen and the application can crash.
Pushing the responsibility of making OpenSSL thread-safe on the
application is a bad design choice and needs to be fixed ASAP to
allow cross-platform, multi-threaded applications to be developed
safely.
________________________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair
______________________________________________________________________
OpenSSL Project
http://www.openssl.org
Development Mailing List
openssl-dev@openssl.org
Automated List Manager
majord...@openssl.org
--
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
web: http://www.hobbelt.com/
http://www.hebbut.net/
mail: g...@hobbelt.com
mobile: +31-6-11 120 978
--------------------------------------------------
________________________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
