Re: [C++-sig] [Boost.Python v3] Conversions and Registries
On 19 Sep 2011 at 17:03, Jim Bosch wrote: > I'd like to see support for static, template-based conversions. These > would be defined by [partial-]specializing a traits class, and I tend to > think they should only be invoked after attempting all registry-based > conversions. Surely not! You'd want to let template specialisaton be the first point of call so the compiler can compile in obvious conversions, *then* and only then do you go to a runtime registry. This also lets one override the runtime registry when needed in the local compiland. I'm not against having another set of template specialisations do something should the first set of specialisations fail, and/or the runtime registry lookup fails. > Users would have to include the same headers in groups of > interdependent modules to avoid specializing the same traits class > multiple times in different ways; I can't think of a way to protect them > from this, but template-based specializations are a sufficiently > advanced featured that I'm comfortable leaving it up to users to avoid > this problem. Just make sure what you do works with precompiled headers :) P.S.: This is trickier than it sounds. > We've had some discussion of allowing different modules to have > different registries (in addition to a global registry shared across > modules). Leaving aside implementation questions, I have a little > survey for interested parties: > > 1) Under what circumstances would you want a conversion that is > completely limited to a specific module (or a group of modules that > explicitly declare it)? Defaults to most recent in calling thread stack, but overridable using a TLS override to allow impersonation. The same mechanism usefully also takes care of multiple python interpreters too. > 2) Under what circumstances would you want a conversion to look in a > module-specific registry, and then fall back to the global registry? As above. That implies that there is no global registry, just the default registry which all module registries inherit. > 3) Considering that we will have a "best-match" overloading system, what > should take precedence, an inexact match in a module-specific registry, > or an exact match in a global registry? (Clearly this is a moot point > for to-Python conversion). The way I've always done this is to have the template metaprogramming set a series of type comparison functions which return scores. This pushes most of the scoring and weighting into the compiler and the compiler will elide any calls into the dynamic registry where the scoring makes that sensible. Makes compile times rather longer though :) The dynamic and compile-time registries can be merged easily enough, so all the runtime registry is is a set of comparison functions normally elided by the compiler in other modules. In other words, mark the inline functions as visible outside the current DLL (dllexport/visibility(default)) so the compiler will assemble complete versions for external usage. > Finally, can anyone give me a reason why having a global registry can > lead to a violation of the "One Definition Rule"? This was alluded to > many times in the earlier discussion, and there's no doubt that a global > registry may lead to unexpected (from a given module's perspective) > behavior - but I do not understand the implication that the global > registry can result in formally undefined behavior by violating the ODR. ODR only matters in practice for anything visible outside the current compiland. If compiling with GCC -fvisibility=hidden, or on any MSVC by default, you can define class foo to be anything you like so long as nothing outside the current compiland can see class foo. ODR is real important though across DLLs. If a DLL X says that class foo is one thing and DLL Y says it's something different, expect things to go very badly wrong. Hence I simply wouldn't have a global registry. It's bad design. You *have* to have per module registries and *only* per module registries. Imagine the following. Program A loads DLL B and DLL C. DLL B is dependent on DLL D which uses BPL. DLL C is dependent on DLL E which uses BPL. DLL D tells BPL that class foo is implicitly convertible with an integer. DLL E tells BPL that class foo is actually a thin wrapper for std::string. Right now with present BPL, we have to load two copies of BPL, one for DLL D and one for DLL E. They maintain separate type registries, so all is good. But what if DLL B returns a python function to Program A, which then installs it as a callback with DLL C? In the normal case, BPL code in DLL E will call into BPL code DLL D and all is well. But what if the function in DLL D throws an exception? This gets converted into a C++ exception by throwing boost::error_already_set. Now the C++ runtime must figure where to send the exception. But what is the C++ runtime supposed to do with such an exception type? It isn't allowed to
Re: [C++-sig] [Boost.Python v3] Conversions and Registries
On 09/20/2011 11:06 AM, Niall Douglas wrote: On 19 Sep 2011 at 17:03, Jim Bosch wrote: I'd like to see support for static, template-based conversions. These would be defined by [partial-]specializing a traits class, and I tend to think they should only be invoked after attempting all registry-based conversions. Surely not! You'd want to let template specialisaton be the first point of call so the compiler can compile in obvious conversions, *then* and only then do you go to a runtime registry. This also lets one override the runtime registry when needed in the local compiland. I'm not against having another set of template specialisations do something should the first set of specialisations fail, and/or the runtime registry lookup fails. I'd also considered having a different set of template conversions that are checked first for performance reasons, but I'd actually viewed the override preference argument from the opposite direction - once a template converter traits class has been fully specialized, you can't specialize it again differently in another module (well, maybe symbol visibility labels can get you out of that bind in practice). So it seemed a registry-based override would be the only way to override a template-based conversion, and hence the registry-based conversions would have to go first. But overall I think your proposal to just try the templates first is cleaner, because having multiple specializations of the same traits class in different modules would be a problem either way; allowing users to override the compile-time conversions with registry-based conversions is at best a poor workaround. Users would have to include the same headers in groups of interdependent modules to avoid specializing the same traits class multiple times in different ways; I can't think of a way to protect them from this, but template-based specializations are a sufficiently advanced featured that I'm comfortable leaving it up to users to avoid this problem. Just make sure what you do works with precompiled headers :) P.S.: This is trickier than it sounds. Yuck. Precompiled headers are something I've never dealt with before, but I suppose I had better learn. We've had some discussion of allowing different modules to have different registries (in addition to a global registry shared across modules). Leaving aside implementation questions, I have a little survey for interested parties: 1) Under what circumstances would you want a conversion that is completely limited to a specific module (or a group of modules that explicitly declare it)? Defaults to most recent in calling thread stack, but overridable using a TLS override to allow impersonation. The same mechanism usefully also takes care of multiple python interpreters too. I have to admit I'm only barely following you here - threads are another thing I don't deal with often. It sounds like you have a totally different option from the ones I was anticipating. Could you explain in more detail how this would work? 2) Under what circumstances would you want a conversion to look in a module-specific registry, and then fall back to the global registry? As above. That implies that there is no global registry, just the default registry which all module registries inherit. (still a little confused about what you mean) 3) Considering that we will have a "best-match" overloading system, what should take precedence, an inexact match in a module-specific registry, or an exact match in a global registry? (Clearly this is a moot point for to-Python conversion). The way I've always done this is to have the template metaprogramming set a series of type comparison functions which return scores. This pushes most of the scoring and weighting into the compiler and the compiler will elide any calls into the dynamic registry where the scoring makes that sensible. Makes compile times rather longer though :) The dynamic and compile-time registries can be merged easily enough, so all the runtime registry is is a set of comparison functions normally elided by the compiler in other modules. In other words, mark the inline functions as visible outside the current DLL (dllexport/visibility(default)) so the compiler will assemble complete versions for external usage. An interesting idea - avoid trying all possible conversions a runtime seems a very worthy goal, though I could also see this inflating the size of the modules. Can you point me at anything existing for an example? Finally, can anyone give me a reason why having a global registry can lead to a violation of the "One Definition Rule"? This was alluded to many times in the earlier discussion, and there's no doubt that a global registry may lead to unexpected (from a given module's perspective) behavior - but I do not understand the implication that the global registry can result in formally undefined behavior by violating the ODR. ODR only matters in practi
Re: [C++-sig] [Boost.Python v3] Conversions and Registries
On 20 Sep 2011 at 12:38, Jim Bosch wrote:
> I'd also considered having a different set of template conversions that
> are checked first for performance reasons, but I'd actually viewed the
> override preference argument from the opposite direction - once a
> template converter traits class has been fully specialized, you can't
> specialize it again differently in another module (well, maybe symbol
> visibility labels can get you out of that bind in practice). So it
> seemed a registry-based override would be the only way to override a
> template-based conversion, and hence the registry-based conversions
> would have to go first.
Ah, sorry, I didn't explain myself well at all. I've been doing a lot
of work surrounding ISO C and C++ standards recently, so my head is
kinda trapped in future C and C++. When I was speaking of ODR, I was
kinda assuming that we have C++ modules available for newer compilers
in the post C++-1x TR (see http://www.open-
std.org/jtc1/sc22/wg21/docs/papers/2007/n2316.pdf) and we can emulate
much of module support using -fvisibility=hidden on GCC. On MSVC, of
course, you get module proto-support for free anyway due to how their
DLLs work.
You're absolutely correct that right now, outside of the Windows
platform, ODR is a process wide problem in most compilers on their
default settings. That's a PITA, so everyone is agreed that we ought
to do something about it. The big problem is how far we ought to go,
hence N2316 not making it into C++-1x and being pushed into TR.
What I can say is that that TR will very likely be highly compatible
with the Windows DLL system (and its GCC visibility near-equivalent)
due to backwards compatibility. I would suggest that you code as if
both are true as a reasonable proxy for future C++ module support.
Then you're covered ten years down the line from now.
> But overall I think your proposal to just try the templates first is
> cleaner, because having multiple specializations of the same traits
> class in different modules would be a problem either way; allowing users
> to override the compile-time conversions with registry-based conversions
> is at best a poor workaround.
I know this is a little off-topic, but Boost could really do with a
generic runtime type registry implementation. There are lots of use
cases outside BPL and if we had one, highly extensible, properly
written system it could be applied to lots of use cases.
For example, Java-style automagical metaprogrammed C++ type
reflection into SQL is perfectly possible. At the time I wrote it, it
was the only example of it anywhere I could find (maybe things have
since changed). It makes talking to databases super-easy at the cost
of making the compiler work very hard.
There are lots more use cases too e.g. talking with .NET, or
Objective C.
> > Just make sure what you do works with precompiled headers :)
> >
> > P.S.: This is trickier than it sounds.
>
> Yuck. Precompiled headers are something I've never dealt with before,
> but I suppose I had better learn.
Getting them working can make the difference between a several hour
recompile and ten minutes. They're painful though due to compiler
bugs.
> > The same mechanism usefully also takes care of multiple python
> > interpreters too.
>
> I have to admit I'm only barely following you here - threads are another
> thing I don't deal with often. It sounds like you have a totally
> different option from the ones I was anticipating. Could you explain in
> more detail how this would work?
Sure. You have the problem when working with Python of handling the
GIL which is strongly related to what the "current" interpreter is.
These are TLS items in python, so each thread has its own current
setting.
Therefore, what one really ought to have in BPL is something like:
// normal C++ code
...
// I want to call python code in interpreter X
{
boost::python::hold_interpreter interpreter_holder(X); // Replaces
"current" interpreter with X
boost::python::hold_GIL gil_holder(interpreter_holder); // Acquire
the GIL for that interpreter
call_some_BPL_or_python_function();
} // On scope exit gil_holder and interpreter_holder gets destroyed,
thus releasing the GIL and resetting the "current" interpreter to
whatever it was before
// Back to normal C++ code
This obviously refers to the embedded case, but it ought to be
similar when BPL calls into C++: the "current" interpreter should be
available per thread as a BPL object instance wrapping the python TLS
config. Then a call into C++ can safely call into other interpreters.
What's useful here of course is that you can keep a per-thread list
of interpreter nestings. This means you can see exactly which module
entered which interpreter and in which order, and therefore what to
search and what to unwind when necessary.
> An interesting idea - avoid trying all possible conversions a runtime
> seems a very worthy goal, though I could also see this
