On 10 Apr 2010 at 21:11, troy d. straszheim wrote:
> I see some special handling in invoke.hpp for
> boost::python::objects::detail::py_iter_, maybe that has something to do
> with it. If one did lock/unlock where I suggest above, one wouldn't
> have the necessary c++ type information to do such handling. Google
> doesn't turn up much on this case... hints?
Yes, I do remember that iterators caught me when I was developing
that patch. I remember being stuck for three or four days tracking
down the bug and getting very frustrated with the metaprogramming
poverty in the MSVC debugger of the time. Not only that, but I also
wanted the future ability to specialise a type to set rules for
locking or unlocking.
> >> Would you also need to lock in e.g. object_protocol.cpp:
> >>
> >> void setattr(object const& target, object const& key, object const& value)
> >> {
> >> if (PyObject_SetAttr(target.ptr(), key.ptr(), value.ptr()) == -1)
> >> throw_error_already_set();
> >> }
> >
> > Maybe I am missing your point, but surely all accesses to Python must
> > hold the GIL first, not least because the GIL also specifies the
> > current interpreter to use? (I know that you can get away with some
> > calls, but relying on this seems chardly xprudent).
>
> Take function new_class(...) in src/object.cpp: this is called during
> BOOST_PYTHON_MODULE(), and invoke.hpp doesn't know about it, therefore
> nothing would be locked.
> [paste]
> During module import that is:
>
> BOOST_PYTHON_MODULE(m)
> {
> class_<T>("T"); // <- here
> }
> [paste]
> Maybe this is a more practical example:
>
> void set_foo_attr(object& obj, object& what)
> {
> api::setattr(obj, "foo", what); // leave cpp in here, no unlock
> }
>
> BOOST_PYTHON_MODULE(m)
> {
> def("set_foo_attr", &set_foo_attr);
> }
>
> >>> class X: pass
> >>> x = X()
> >>> set_foo_attr(x, ['something'])
> >>> print x.foo
> ['something']
>
>
> Here, enter/exitCPP would be called by invoke.hpp when set_foo_attr
> fires, but not when api::setattr calls PyObject_SetAttrString.
Ah I see your point now.
TnFOX's BPL patch made the assumption that when python calls into C++
land that it was going to be executing C++ stuff rather than
interfacing with C++ representations of python stuff. This was a very
reasonable assumption to make when you are simply providing python
bindings for a large C++ library, so on the point of entering C++
land it unlocks the GIL and sets the current python interpreter to
null. For that thread, any attempt to do *anything* with python
henceforth guarantees an exception.
Now if that C++ thread happens to want to do some work in Python, it
must hold a RAII instance of FXPython_CtxHold around the relevant
code. FXPython_CtxHold simply takes a FXPythonInterp to set as
current - this grabs the GIL and sets the current python interpreter.
I have never been particularly happy with this solution because
excess locking is real bad for system latency i.e. when python calls
a virtual function the GIL gets unlocked, relocked, a check for a
python override is made, unlocked, jump to base class implementation,
on exit relock. I personally would use a technique I call "a hanging
lock" whereby you wrap a parameter with a thin metaprogramming
construct which lets you effectively bind a RAII held lock with a
parameter such that a callee takes possession, but I don't know if
this is a common technique in Boost and adding it made patching in
updates too difficult.
> > Hence it may well be that a static signals and slots implementation
> > could be more appropriate in this situation. I guess I wouldn't know
> > until I run benchmarks. Your thoughts?
>
> Thanks for the discussion. I'm now thinking that the handling of these
> enter/exit "signals" emitted by boost.python shouldn't be coupled to
> boost::signals or anything else. Seems cleaner and easier to provide an
> interface behind which one could put a simple lock/unlocker or something
> more complicated involving boost::signals if desired.
I was maybe thinking of doing it properly with a boost::staticsignals
library.
> [snip]
> > In TnFOX I have a metaprogramming construct which assembles
inline a
> > jump table of specialisations of classes between which at run
time
> > can be dynamically chosen. Fairly amazingly, all major compilers
> > correctly elide table entries which cannot be chosen such that
they
> > will remove the choice logic entirely if there is just one
possible
> > choice, or the whole construct if there are none. This particular
> > construct is really useful in BPL actually because it lets you
fake
> > stuff like setting at runtime arbitrary python code as a C (not
C++)
> > API based sort function.
>
> Could you point me at the code?
Sure.
The general TnFOX docs are at http://tnfox.sourceforge.net/TnFOX-
svn/html/ where the header file FXGenericTools.h
(http://tnfox.sourceforge.net/TnFOX-
svn/html/_f_x_generic_tools_8h.html) contains most of the
metaprogramming library. In this file there is a template called
FX::Generic::TL::dynamicAt< typelist, instance >
(http://tnfox.sourceforge.net/TnFOX-
svn/html/struct_f_x_1_1_generic_1_1_t_l_1_1dynamic_at.html). You can
see the source of FXGenericTools.h at
http://tnfox.sourceforge.net/TnFOX-svn/html/_f_x_generic_tools_8h-
source.html with dynamicAt<> being at around line 2226. As you'll
note, dynamicAt<> statically assembles a sixteen entry array of
runtime generated code points into read only space, then it jumps
into it by array indexation. I have to hand full kudos to the
compiler guys such that the compiler knows the right thing to do if
the array index is fixed.
The example I refered to of faking arbitrary python code is
documented at http://tnfox.sourceforge.net/TnFOX-
svn/html/class_f_x_1_1_f_x_code_to_python_code.html.
It may help you when you are reading the code to know that
FX::Generic::TL::instantiateH<> is a horizontal instantiator of a
typelist i.e. TL::instantiateH<TL::create<A, B, C>::value, Holder>
will make a class instantiateH<> : public Holder<A>, public
Holder<B>, public Holder<C>.
If you have any questions then please do let me know - I appreciate
that yet another metaprogramming library can hurt the head. With
regard to implementing support for all this, could you clarify your
planned timetable with respect to your branch of Boost?
Cheers,
Niall
_______________________________________________
Cplusplus-sig mailing list
Cplusplus-sig@python.org
http://mail.python.org/mailman/listinfo/cplusplus-sig
