Hi Dane, Thanks for the feedback. If there are no other objections and if no one else gets to it first, I can look through the build scripts to try to find this. I've never used scons before, but its just Python, right :)
I may not get to it right away though because I am trying to land the work I was doing prior to stumbling on this. Mapnik has come a long way in a short period of time. The third party evolution of language bindings is a testament to that. I am interested in the opinion of whether these eventually go into the core mapnik distro or not. Purely from the perspective of conserving size, baking the JNI bindings into the core would save a fair chunk of compiled size. Currently the Linux x86_64 JNI library is about 650KB and isn't fully implemented. I think this is due to the fact that all of the inline code and templates get instantiated in each shared library as weak symbols, just to be ignored later. The actual unique code that goes into the binding is pretty small. It's also pretty straightforward to build: Just one CPP file and a handful of compiler flags that are determined by java-config. It would be pretty easy to drop into the core distribution as an optional component to be built if Java headers are present, but I'm going back and forth on my own opinion as to whether it should be in core or separate. - stella On Tue, Jun 28, 2011 at 9:56 AM, Dane Springmeyer <[email protected]> wrote: > Hi Stella, > > Great to hear from you. Comments below... > > On Jun 27, 2011, at 11:52 PM, Stella Laurenzo wrote: > > Hi Mapnik developers, > In conjunction with a project I'm working on, I've been building out some > JNI bindings for mapnik. They are here: > https://github.com/stellaeof/mapnik-jni > > At some point, I'd like to discuss merging this into the core mapnik > project where I think such things belong, but for now, I've stumbled across > a minor mystery. The bindings work perfectly on OSX and fail miserably on > Linux. I'm so used to shared library weirdness on OSX, that I was a little > bit surprised by this since Linux is usually the well behaved citizen in > this regard. > > After an evening of digging I ran across several related threads. The most > pertinent is here: > http://gis.638310.n2.nabble.com/libtool-Error-when-loading-input-plugins-from-shared-library-td5176657.html > but > it was never resolved. > > The issue specifically is that I have a shared library libmapnik-jni.so > that contains the JNI exports and depends on libmapnik2.so. Everything > works fine until attempting to load an input plugin. Upon calling > datasource_cache::register_datasources(...), the following type of message > is written to stderr: > > Problem loading plugin library: /usr/local/lib/mapnik2/input/sqlite.input > (dlopen failed - plugin likely has an unsatisfied dependency or incompatible > ABI) > > Then, when calling datasource_cache::create with parameters referencing a > postgis plugin, the following is written and the process crashes: > > /usr/lib/jvm/java-6-openjdk/bin/java: symbol lookup error: > /usr/local/lib/mapnik2/input/postgis.input: undefined symbol: > _ZN6mapnik5box2dIdEC1Ev > > > Depending on the load order and such different messages and failures can > occur but these are the gist of it. > > While tracking this down, the first thing I noticed was that if I defined > LD_PRELOAD=/usr/local/lib/libmapnik2.so prior to running the Java process > that does all of this, everything works. This led me to look at the plugins > themselves and I noticed that the input plugins do not have a library > dependency on libmapnik2.so. On OSX they do because things can't be left > partially linked on that platform, but on Linux mapnik must be assuming that > in order for a plugin to load, the libmapnik2.so library must already be > loaded globally. To test this, I ran the script here in the plugin > directory: > https://github.com/stellaeof/mapnik-jni/blob/master/fixup-input-plugins.sh > > This just renames each .input file and links a new one that depends on the > original and libmapnik2.so. With this done, everything works as it should > and no warnings are printed when the plugins are loaded. > > I believe that the issue is that on the JVM, when a library is loaded (via > System.load or its cousin System.loadLibrary), dlopen is being called on the > library without the RTLD_GLOBAL flag. This is probably a good thing as it > keeps name collisions to a minimum in JNI libraries. However, it means that > when libmapnik2.so goes to dlopen() its input plugins, it results in > undefined symbols because even though the libmapnik2.so library is resident > in the process, its symbols have not been exported for all to use. > > If I were to file a bug against mapnik, it would be titled "Input plugins > fail to load when libmapnik2.so loaded via dlopen call without RTLD_GLOBAL > flag". There are several solutions: > - Include libmapnik2.so as a dependency in mapnik input plugins, similar > to how my script fixes things up to be after the fact. This would mirror > OSX behavior and I do not believe it would have any deleterious effects. > - Declare that libmapnik2.so can only be used when explicitly linked to > or via a dlopen(RTLD_GLOBAL) call. This would mean that it will never work > when loaded directly or indirectly by standard JVMs. > - Leave it to someone else (me). In this case, I suppose I could dlopen > the JNI library again with the RTLD_GLOBAL flag prior to loading any > plugins. The man page says this should work. Of course, I'd have to find > the full path to the shared library, which I don't necessarily know, and > this introduces more failure paths and is a little bit crazy. > > Unless if anyone is aware of some issue that I don't know about, I think > the first option is the best. It would mean that mapnik would always be > able to load its input plugins regardless of how itself was loaded. I'm not > at all familiar with the mapnik build process to know where to make this > change, however. > > - stella > > > > RE: JNI bindings in general - wow. As little as a year ago the Mapnik > project just had Python bindings. Now we have emerging bindings for PHP [1], > NodeJS [2], Ruby [3], and Java. So, its a good time to re-assess and discuss > both how development of bindings can be optimal (inside/outside core, > release cycle, etc), and any changes we need to make in core to make life > good for various bindings. > > RE: plugin linking, your analysis is spot on. Of your three solutions, I > also agree that we should likely move to #1 - actually linking any required > dependencies such that plugins can be loaded without having to monkey with > dlopen flags. The reason we have resisted doing this so far is that Python > has an easy mechanism to control dlopen flags [4]. But, I noticed the same > problem as you hit with NodeJS recently [5]. My workaround on linux is to > dlopen libmapnik.so first [6], which of course is an ugly hack. But it works > well enough I've forgotten to circle back on this issue, till now. > > So, if we go with solution #1, to make Linux linking like OSX, the solution > is pretty easy. We just need to find all the linking lines in each plugin > build file that are specific to Darwin and make them apply no matter > platform (we only currently support linux, freebsd, solaris, and osx with > the SCons scripts). > > An example of that would be here: > > http://trac.mapnik.org/browser/trunk/plugins/input/shape/SConscript#L42 > > Artem, others hav comments/drawbacks of this on Linux? I know Fedora > recently changed their policy to be more like OSX. I can't see any drawbacks > of this approach. It just needs a bit of testing I think. > > Dane > > [1] https://github.com/roel-v/php_mapnik > [2] https://github.com/mapnik/node-mapnik > [3] https://github.com/elliotlaster/Pika > [4] > http://trac.mapnik.org/browser/trunk/bindings/python/mapnik/__init__.py#L55 > [5] https://github.com/joyent/node/issues/436 > [6] https://github.com/mapnik/node-mapnik/blob/master/lib/mapnik.js#L31-44 > >
_______________________________________________ Mapnik-devel mailing list [email protected] https://lists.berlios.de/mailman/listinfo/mapnik-devel
