Neil, great information, but I did track this problem down to a quirk with Py++.
I've had a great deal of trouble finding a reliable way to actually write
modules with module_builder. Originally, I had been using split_module but
I've run into several cases where it goes off in the weeds and tries to write
files that exceed the maximum file length. Generally, this occurs when
wrapping classes that go nuts on specialization since module_builder uses the
name of the class in the file bindings file. Here is one quick, extreme
example where this occurs when trying to wrap members of boost:
File "C:\Users\davidsj2\workspace\SimCommon\src\Python\goes\build\bindings.py",
line 325, in _generate
files = mb.split_module(self._bindingsDir)
File
"c:\Python26\lib\site-packages\pyplusplus\module_builder\boost_python_builder.py",
line 375, in split_module
, encoding=self.encoding)
File "c:\Python26\lib\site-packages\pyplusplus\file_writers\__init__.py",
line 37, in write_multiple_files
mfs.write()
File
"c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line
406, in write
self.split_classes()
File
"c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line
307, in split_classes
map( self.split_class, class_creators )
File
"c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line
294, in split_class
self.split_class_impl( class_creator )
File
"c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line
268, in split_class_impl
, self.create_function_code( function_name ) ) )
File
"c:\Python26\lib\site-packages\pyplusplus\file_writers\multiple_files.py", line
61, in write_file
writer.writer_t.write_file( fpath, content, self.files_sum_repository,
self.encoding )
File "c:\Python26\lib\site-packages\pyplusplus\file_writers\writer.py", line
150, in write_file
f = codecs.open( fpath, 'w+b', encoding )
File "c:\Python26\lib\codecs.py", line 881, in open
file = __builtin__.open(filename, mode, buffering)
IOError: [Errno 2] No such file or directory:
'C:\\Users\\davidsj2\\workspace\\SimCommon\\build\\win64\\pybindings\\goes\\boost\\dividable2_less__boost_scope_date_time_scope_date_duration_less__boost_scope_date_time_scope_duration_traits_adapted__greater__comma__int_comma__boost_scope_detail_scope_empty_base_less__boost_scope_date_time_scope_date_duration_less__boost_scope_date_time_scope_duration_traits_adapted__greater___greater___greater_.pypp.hpp'
make: *** [all] Error 1
After finding references to this problem as far back as 2006, I decided to
switch over to balanced_split_module. This has its own set of problems. The
first is that it is highly prone to divide by zero errors. One quick way to
reproduce this issue is to wrap one class and specify a split count of 2.
Obviously not a wise combo, but it's an easy error case that Py++ should handle.
So anyways, the root of *this* problem is how balanced_split_module creates its
registration functions. For each extension, it creates one register function
for each file it writes in the form: void register_classes_()Obviously,
these collide when you create more than one extension using
balanced_split_module and enable RTLD_GLOBAL. One quick solution to this
problem would be to prepend the extension name to the name of the registration
functions, e.g.: _register_classes_ Since the module name is used
to name the files, its easily accessible and would solve a lot of problems. Of
course, if you have modules with the same name in different packages you would
run into this again.
Josh
-Original Message-
From: cplusplus-sig-bounces+josh.davidson=lmco@python.org
[mailto:cplusplus-sig-bounces+josh.davidson=lmco@python.org] On Behalf Of
Niall Douglas
Sent: Wednesday, February 01, 2012 10:46 AM
To: Development of Python/C++ integration
Subject: Re: [C++-sig] EXTERNAL: Re: Odd dlopen behavior
On 31 Jan 2012 at 16:44, Davidson, Josh wrote:
> Ok, well I did figure out the discrepancy between these extensions and
> previous extensions that have been built that required setting
> RTLD_GLOBAL. What I'm doing for these extensions is instead of
> building in all of the original C++ code AND the Py++ generated code
> into the extension, I'm only building in the Py++ generated sources
> and linking an existing shared library containing the original C++
> definitions. Is this non-standard or bad practice?
The big problem with shared objects exporting lots of symbols was that the
Linux runtime shared object linker used to have O(N^3) complexity. As a result,
every time you ran a program linking to a ginormous shared object you'd get a
pause of several seconds as it bound the symbols.
Now, some years ago with KDE and OpenOffice taking forever to load, some
eyeballs were turned onto this problem and I know they were going to get it
down to O(N^2). There was speak of replacin
