Quick correction to my previous mail: The Linux x86_64 library is now about 261KB. Something I did last night must have changed that dramatically... not quite sure what.
- stella On Tue, Jun 28, 2011 at 11:16 AM, Stella Laurenzo <[email protected]>wrote: > Hi Dane, > Thanks for the feedback. If there are no other objections and if no one > else gets to it first, I can look through the build scripts to try to find > this. I've never used scons before, but its just Python, right :) > > I may not get to it right away though because I am trying to land the work > I was doing prior to stumbling on this. > > Mapnik has come a long way in a short period of time. The third party > evolution of language bindings is a testament to that. I am interested in > the opinion of whether these eventually go into the core mapnik distro or > not. Purely from the perspective of conserving size, baking the JNI > bindings into the core would save a fair chunk of compiled size. Currently > the Linux x86_64 JNI library is about 650KB and isn't fully implemented. I > think this is due to the fact that all of the inline code and templates get > instantiated in each shared library as weak symbols, just to be ignored > later. The actual unique code that goes into the binding is pretty small. > It's also pretty straightforward to build: Just one CPP file and a handful > of compiler flags that are determined by java-config. It would be pretty > easy to drop into the core distribution as an optional component to be built > if Java headers are present, but I'm going back and forth on my own opinion > as to whether it should be in core or separate. > > - stella > > > On Tue, Jun 28, 2011 at 9:56 AM, Dane Springmeyer <[email protected]> wrote: > >> Hi Stella, >> >> Great to hear from you. Comments below... >> >> On Jun 27, 2011, at 11:52 PM, Stella Laurenzo wrote: >> >> Hi Mapnik developers, >> In conjunction with a project I'm working on, I've been building out some >> JNI bindings for mapnik. They are here: >> https://github.com/stellaeof/mapnik-jni >> >> At some point, I'd like to discuss merging this into the core mapnik >> project where I think such things belong, but for now, I've stumbled across >> a minor mystery. The bindings work perfectly on OSX and fail miserably on >> Linux. I'm so used to shared library weirdness on OSX, that I was a little >> bit surprised by this since Linux is usually the well behaved citizen in >> this regard. >> >> After an evening of digging I ran across several related threads. The >> most pertinent is here: >> http://gis.638310.n2.nabble.com/libtool-Error-when-loading-input-plugins-from-shared-library-td5176657.html >> but >> it was never resolved. >> >> The issue specifically is that I have a shared library libmapnik-jni.so >> that contains the JNI exports and depends on libmapnik2.so. Everything >> works fine until attempting to load an input plugin. Upon calling >> datasource_cache::register_datasources(...), the following type of message >> is written to stderr: >> >> Problem loading plugin library: /usr/local/lib/mapnik2/input/sqlite.input >> (dlopen failed - plugin likely has an unsatisfied dependency or incompatible >> ABI) >> >> Then, when calling datasource_cache::create with parameters referencing a >> postgis plugin, the following is written and the process crashes: >> >> /usr/lib/jvm/java-6-openjdk/bin/java: symbol lookup error: >> /usr/local/lib/mapnik2/input/postgis.input: undefined symbol: >> _ZN6mapnik5box2dIdEC1Ev >> >> >> Depending on the load order and such different messages and failures can >> occur but these are the gist of it. >> >> While tracking this down, the first thing I noticed was that if I defined >> LD_PRELOAD=/usr/local/lib/libmapnik2.so prior to running the Java process >> that does all of this, everything works. This led me to look at the plugins >> themselves and I noticed that the input plugins do not have a library >> dependency on libmapnik2.so. On OSX they do because things can't be left >> partially linked on that platform, but on Linux mapnik must be assuming that >> in order for a plugin to load, the libmapnik2.so library must already be >> loaded globally. To test this, I ran the script here in the plugin >> directory: >> https://github.com/stellaeof/mapnik-jni/blob/master/fixup-input-plugins.sh >> >> This just renames each .input file and links a new one that depends on the >> original and libmapnik2.so. With this done, everything works as it should >> and no warnings are printed when the plugins are loaded. >> >> I believe that the issue is that on the JVM, when a library is loaded (via >> System.load or its cousin System.loadLibrary), dlopen is being called on the >> library without the RTLD_GLOBAL flag. This is probably a good thing as it >> keeps name collisions to a minimum in JNI libraries. However, it means that >> when libmapnik2.so goes to dlopen() its input plugins, it results in >> undefined symbols because even though the libmapnik2.so library is resident >> in the process, its symbols have not been exported for all to use. >> >> If I were to file a bug against mapnik, it would be titled "Input plugins >> fail to load when libmapnik2.so loaded via dlopen call without RTLD_GLOBAL >> flag". There are several solutions: >> - Include libmapnik2.so as a dependency in mapnik input plugins, >> similar to how my script fixes things up to be after the fact. This would >> mirror OSX behavior and I do not believe it would have >> any deleterious effects. >> - Declare that libmapnik2.so can only be used when explicitly linked to >> or via a dlopen(RTLD_GLOBAL) call. This would mean that it will never work >> when loaded directly or indirectly by standard JVMs. >> - Leave it to someone else (me). In this case, I suppose I could >> dlopen the JNI library again with the RTLD_GLOBAL flag prior to loading any >> plugins. The man page says this should work. Of course, I'd have to find >> the full path to the shared library, which I don't necessarily know, and >> this introduces more failure paths and is a little bit crazy. >> >> Unless if anyone is aware of some issue that I don't know about, I think >> the first option is the best. It would mean that mapnik would always be >> able to load its input plugins regardless of how itself was loaded. I'm not >> at all familiar with the mapnik build process to know where to make this >> change, however. >> >> - stella >> >> >> >> RE: JNI bindings in general - wow. As little as a year ago the Mapnik >> project just had Python bindings. Now we have emerging bindings for PHP [1], >> NodeJS [2], Ruby [3], and Java. So, its a good time to re-assess and discuss >> both how development of bindings can be optimal (inside/outside core, >> release cycle, etc), and any changes we need to make in core to make life >> good for various bindings. >> >> RE: plugin linking, your analysis is spot on. Of your three solutions, I >> also agree that we should likely move to #1 - actually linking any required >> dependencies such that plugins can be loaded without having to monkey with >> dlopen flags. The reason we have resisted doing this so far is that Python >> has an easy mechanism to control dlopen flags [4]. But, I noticed the same >> problem as you hit with NodeJS recently [5]. My workaround on linux is to >> dlopen libmapnik.so first [6], which of course is an ugly hack. But it works >> well enough I've forgotten to circle back on this issue, till now. >> >> So, if we go with solution #1, to make Linux linking like OSX, the >> solution is pretty easy. We just need to find all the linking lines in each >> plugin build file that are specific to Darwin and make them apply no matter >> platform (we only currently support linux, freebsd, solaris, and osx with >> the SCons scripts). >> >> An example of that would be here: >> >> http://trac.mapnik.org/browser/trunk/plugins/input/shape/SConscript#L42 >> >> Artem, others hav comments/drawbacks of this on Linux? I know Fedora >> recently changed their policy to be more like OSX. I can't see any drawbacks >> of this approach. It just needs a bit of testing I think. >> >> Dane >> >> [1] https://github.com/roel-v/php_mapnik >> [2] https://github.com/mapnik/node-mapnik >> [3] https://github.com/elliotlaster/Pika >> [4] >> http://trac.mapnik.org/browser/trunk/bindings/python/mapnik/__init__.py#L55 >> [5] https://github.com/joyent/node/issues/436 >> [6] >> https://github.com/mapnik/node-mapnik/blob/master/lib/mapnik.js#L31-44 >> >> >
_______________________________________________ Mapnik-devel mailing list [email protected] https://lists.berlios.de/mailman/listinfo/mapnik-devel
