Hi Stella, Great to hear from you. Comments below...
On Jun 27, 2011, at 11:52 PM, Stella Laurenzo wrote: > Hi Mapnik developers, > In conjunction with a project I'm working on, I've been building out some JNI > bindings for mapnik. They are here: https://github.com/stellaeof/mapnik-jni > > At some point, I'd like to discuss merging this into the core mapnik project > where I think such things belong, but for now, I've stumbled across a minor > mystery. The bindings work perfectly on OSX and fail miserably on Linux. > I'm so used to shared library weirdness on OSX, that I was a little bit > surprised by this since Linux is usually the well behaved citizen in this > regard. > > After an evening of digging I ran across several related threads. The most > pertinent is here: > http://gis.638310.n2.nabble.com/libtool-Error-when-loading-input-plugins-from-shared-library-td5176657.html > but it was never resolved. > > The issue specifically is that I have a shared library libmapnik-jni.so that > contains the JNI exports and depends on libmapnik2.so. Everything works fine > until attempting to load an input plugin. Upon calling > datasource_cache::register_datasources(...), the following type of message is > written to stderr: > Problem loading plugin library: /usr/local/lib/mapnik2/input/sqlite.input > (dlopen failed - plugin likely has an unsatisfied dependency or incompatible > ABI) > Then, when calling datasource_cache::create with parameters referencing a > postgis plugin, the following is written and the process crashes: > /usr/lib/jvm/java-6-openjdk/bin/java: symbol lookup error: > /usr/local/lib/mapnik2/input/postgis.input: undefined symbol: > _ZN6mapnik5box2dIdEC1Ev > > Depending on the load order and such different messages and failures can > occur but these are the gist of it. > > While tracking this down, the first thing I noticed was that if I defined > LD_PRELOAD=/usr/local/lib/libmapnik2.so prior to running the Java process > that does all of this, everything works. This led me to look at the plugins > themselves and I noticed that the input plugins do not have a library > dependency on libmapnik2.so. On OSX they do because things can't be left > partially linked on that platform, but on Linux mapnik must be assuming that > in order for a plugin to load, the libmapnik2.so library must already be > loaded globally. To test this, I ran the script here in the plugin > directory: > https://github.com/stellaeof/mapnik-jni/blob/master/fixup-input-plugins.sh > > This just renames each .input file and links a new one that depends on the > original and libmapnik2.so. With this done, everything works as it should > and no warnings are printed when the plugins are loaded. > > I believe that the issue is that on the JVM, when a library is loaded (via > System.load or its cousin System.loadLibrary), dlopen is being called on the > library without the RTLD_GLOBAL flag. This is probably a good thing as it > keeps name collisions to a minimum in JNI libraries. However, it means that > when libmapnik2.so goes to dlopen() its input plugins, it results in > undefined symbols because even though the libmapnik2.so library is resident > in the process, its symbols have not been exported for all to use. > > If I were to file a bug against mapnik, it would be titled "Input plugins > fail to load when libmapnik2.so loaded via dlopen call without RTLD_GLOBAL > flag". There are several solutions: > - Include libmapnik2.so as a dependency in mapnik input plugins, similar > to how my script fixes things up to be after the fact. This would mirror OSX > behavior and I do not believe it would have any deleterious effects. > - Declare that libmapnik2.so can only be used when explicitly linked to or > via a dlopen(RTLD_GLOBAL) call. This would mean that it will never work when > loaded directly or indirectly by standard JVMs. > - Leave it to someone else (me). In this case, I suppose I could dlopen > the JNI library again with the RTLD_GLOBAL flag prior to loading any plugins. > The man page says this should work. Of course, I'd have to find the full > path to the shared library, which I don't necessarily know, and this > introduces more failure paths and is a little bit crazy. > > Unless if anyone is aware of some issue that I don't know about, I think the > first option is the best. It would mean that mapnik would always be able to > load its input plugins regardless of how itself was loaded. I'm not at all > familiar with the mapnik build process to know where to make this change, > however. > > - stella RE: JNI bindings in general - wow. As little as a year ago the Mapnik project just had Python bindings. Now we have emerging bindings for PHP [1], NodeJS [2], Ruby [3], and Java. So, its a good time to re-assess and discuss both how development of bindings can be optimal (inside/outside core, release cycle, etc), and any changes we need to make in core to make life good for various bindings. RE: plugin linking, your analysis is spot on. Of your three solutions, I also agree that we should likely move to #1 - actually linking any required dependencies such that plugins can be loaded without having to monkey with dlopen flags. The reason we have resisted doing this so far is that Python has an easy mechanism to control dlopen flags [4]. But, I noticed the same problem as you hit with NodeJS recently [5]. My workaround on linux is to dlopen libmapnik.so first [6], which of course is an ugly hack. But it works well enough I've forgotten to circle back on this issue, till now. So, if we go with solution #1, to make Linux linking like OSX, the solution is pretty easy. We just need to find all the linking lines in each plugin build file that are specific to Darwin and make them apply no matter platform (we only currently support linux, freebsd, solaris, and osx with the SCons scripts). An example of that would be here: http://trac.mapnik.org/browser/trunk/plugins/input/shape/SConscript#L42 Artem, others hav comments/drawbacks of this on Linux? I know Fedora recently changed their policy to be more like OSX. I can't see any drawbacks of this approach. It just needs a bit of testing I think. Dane [1] https://github.com/roel-v/php_mapnik [2] https://github.com/mapnik/node-mapnik [3] https://github.com/elliotlaster/Pika [4] http://trac.mapnik.org/browser/trunk/bindings/python/mapnik/__init__.py#L55 [5] https://github.com/joyent/node/issues/436 [6] https://github.com/mapnik/node-mapnik/blob/master/lib/mapnik.js#L31-44
_______________________________________________ Mapnik-devel mailing list [email protected] https://lists.berlios.de/mailman/listinfo/mapnik-devel
