Re: suggestions for handling native libraries
Mark Donszelmann wrote: Hi in the freehep-nar-plugin (for maven 1, being ported to maven 2) we have such a solution. snip Hrm, I wish I'd heard about this sooner! Sounds very interesting. I have a couple of questions... Do you unpack all nar files into the same directory? If so, how do you deal with different projects needing different versions of nars? Cheers, Rich -- Richard van der Hoff [EMAIL PROTECTED] Telephony Gateways Project Manager Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
The jar file and its attached nar files share one pom, so are associated to one version. The NAR gets unpacked in the local repository under the directory where the jar file is stored, so underneath a parent directory with that version number. Nothing is currently done about locking the file system while unpacking, so running two mavens in parallel using the same local repository may not work correctly, but I have seen maven (with jars only) making mistakes in writing metafiles when two mavens run in parallel, so this may be a general problem still. Regards Mark On Jun 19, 2006, at 2:32 AM, Richard van der Hoff wrote: Mark Donszelmann wrote: Hi in the freehep-nar-plugin (for maven 1, being ported to maven 2) we have such a solution. snip Hrm, I wish I'd heard about this sooner! Sounds very interesting. I have a couple of questions... Do you unpack all nar files into the same directory? If so, how do you deal with different projects needing different versions of nars? Cheers, Rich -- Richard van der Hoff [EMAIL PROTECTED] Telephony Gateways Project Manager Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
Mark Donszelmann wrote: The NAR gets unpacked in the local repository under the directory where the jar file is stored, so underneath a parent directory with that version number. Ah ha, I see. In that case, how does your System.load() call know where to find the library? I guess it has knowledge of group, artifact, version built into the java jar? And if you have libraries which are linked at the native level, how does the os library loader know where to find the prerequisite libraries? Cheers, Richard -- Richard van der Hoff [EMAIL PROTECTED] Telephony Gateways Project Manager Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
Hi System.load: - for the user we still need to make some assembly goal that grabs all the shared and jni like libs and puts them in a lib directory. A small setup script can then set the appropriate ld_library_path or the like. - for maven the nar plugin includes an integration test goal, which runs after packaging, and runs as part of maven, so it knows how to set the java.library.path to run the tests. The lava.library.path is set to the produced jni/ so file and all its dependencies. Naming: The artifactID and version are picked up from the standard maven properties file in the jar file. Regards Mark On Jun 19, 2006, at 8:18 AM, Richard van der Hoff wrote: Mark Donszelmann wrote: The NAR gets unpacked in the local repository under the directory where the jar file is stored, so underneath a parent directory with that version number. Ah ha, I see. In that case, how does your System.load() call know where to find the library? I guess it has knowledge of group, artifact, version built into the java jar? And if you have libraries which are linked at the native level, how does the os library loader know where to find the prerequisite libraries? Cheers, Richard -- Richard van der Hoff [EMAIL PROTECTED] Telephony Gateways Project Manager Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
Mark Donszelmann wrote: Hi System.load: - for the user we still need to make some assembly goal that grabs all the shared and jni like libs and puts them in a lib directory. A small setup script can then set the appropriate ld_library_path or the like. - for maven the nar plugin includes an integration test goal, which runs after packaging, and runs as part of maven, so it knows how to set the java.library.path to run the tests. The lava.library.path is set to the produced jni/so file and all its dependencies. I see. Well, this has all been very interesting - thanks Mark! It seems that we've both considered the same set of problems, and come up with slightly different solutions. I only wish I'd heard about this a month ago - I'd much rather have worked with you on the nar plugin than made my own version. I'll watch development of your plugin with interest :). Cheers, Richard -- Richard van der Hoff [EMAIL PROTECTED] Telephony Gateways Project Manager Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
On 5/9/06, Richard van der Hoff [EMAIL PROTECTED] wrote: ... This brings me to my next thought, which is that the maven artifact probably ought to be a zip file (or similar) containing the correctly-named library. This is good because it means my artifacts have the same extension no matter what platform they were built for; I can distinguish between platforms using the classifier; and I can be sure that the name of the library itself is just right for the given platform. On the other hand, it means I now need to set up my assembly such that the zip files are unpacked. Ideally, I'd like for all such artifacts to be unpacked automatically, but there doesn't seem to be any way of doing so without configuring it in each assembly.xml. It's a property of the artifact, rather than the assembly, so it would be nice to somehow derive this from an ArtifactHandler rather than in the assembly descriptor. I can probably fudge this with a shared assembly descriptor, however. ... I think your on the right path here. Having the native artifacts in zip would very handy. But how about instead of the artifact being a zip, it's a directory? If maven could deploy and retrieve a whole directory as an artifact, it should make things easier for your use case correct? The reason I'm think this way, is that in the native world, if you want to distribute a native .dll (shared library), the library is more than just the dll, you also have to distribute the .h files the define the interfaces into that binary shared library. And if there were being pushed up into a maven repo, then when it was downloaded, the c++ compiler would need to add to the include path the directory where those .h files are at. So those .h files need to be in a directory, not in a .zip file. And note, if an artifact is a directory, I don't care is maven zip / tars that directory when it puts it on the remote repo, a long as when it puts it on the local repo it back to it's original directory form. -- Regards, Hiram Blog: http://hiramchirino.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
Hi in the freehep-nar-plugin (for maven 1, being ported to maven 2) we have such a solution. We build a generic jar file (with classes as needed and a special properties file). The properties file refers to native jar files (nar files), normally: the noarch file which has the include directories and a aol (architecture-os-linker) file which contains native libraries. The user in his pom just refers to the generic jar file for his dependency. The NAR plugin uses (for maven 2) its own packaging (nar) which resembles the jar packaging, but adds goals to several phases. Maven will download the generic jar file. The nar packing will look in the properties file and download any referred nar files (some of which are aol specific for the platform you run maven on). It will then unpack the nar files into a nar directory in the local repository. And add specific include paths and library settings to the local compiler/linker setup. This uses cpptasks to get the job done. In the end, the generated lib and include files for the project are wrapped up in a generic jar file and a noarch and aol nar file. These are installed/deployed as usual. The noarch and aol files are attached artifacts. Another project can thus rely on this (native) library. There is very little to configure by the user, apart from a reference to the nar plugin, the type (shared, static, ...) of library to be produced, and the nar packaging. The nar plugin exists for maven 1 (with a slightly different approach since there is no packaging in maven1) http://java.freehep.org/freehep-nar-plugin and we are porting it currently to maven2. Regards Mark Donszelmann On Jun 16, 2006, at 10:32 AM, Hiram Chirino wrote: On 5/9/06, Richard van der Hoff [EMAIL PROTECTED] wrote: ... This brings me to my next thought, which is that the maven artifact probably ought to be a zip file (or similar) containing the correctly-named library. This is good because it means my artifacts have the same extension no matter what platform they were built for; I can distinguish between platforms using the classifier; and I can be sure that the name of the library itself is just right for the given platform. On the other hand, it means I now need to set up my assembly such that the zip files are unpacked. Ideally, I'd like for all such artifacts to be unpacked automatically, but there doesn't seem to be any way of doing so without configuring it in each assembly.xml. It's a property of the artifact, rather than the assembly, so it would be nice to somehow derive this from an ArtifactHandler rather than in the assembly descriptor. I can probably fudge this with a shared assembly descriptor, however. ... I think your on the right path here. Having the native artifacts in zip would very handy. But how about instead of the artifact being a zip, it's a directory? If maven could deploy and retrieve a whole directory as an artifact, it should make things easier for your use case correct? The reason I'm think this way, is that in the native world, if you want to distribute a native .dll (shared library), the library is more than just the dll, you also have to distribute the .h files the define the interfaces into that binary shared library. And if there were being pushed up into a maven repo, then when it was downloaded, the c++ compiler would need to add to the include path the directory where those .h files are at. So those .h files need to be in a directory, not in a .zip file. And note, if an artifact is a directory, I don't care is maven zip / tars that directory when it puts it on the remote repo, a long as when it puts it on the local repo it back to it's original directory form. -- Regards, Hiram Blog: http://hiramchirino.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
Hi Dan,
dan tran wrote:
Hi Richard,
The native-maven-plugin JNI example shows that you can have your native
project depends on a java jar project that has
java jni interfaces. This way you can use native plugin to generate jni
native include files before building the native artifacts.
This is the same behavior in maven1.
Not sure why it is a blocking to you since it works
The question is, how do you then arrange for another project (maybe with
more JNI, maybe not) to depend on that {native,java} project pair? It
all gets incredibly messy. And believe me, I tried.
Furthermore, what do you do about unit tests with the
native-maven-plugin? They have to run after the native artifacts are
built, obviously.
I don't mean to step on your toes by setting up an alternative to the
native-maven-plugin; it's just that the native-maven-plugin really
didn't suit my needs at all.
Cheers,
Richard
--
Richard van der Hoff [EMAIL PROTECTED]
Telephony Gateways Project Manager
Tel: +44 (0) 845 666 7778
http://www.mxtelecom.com
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
A while ago, I posted here asking for help with using Maven to handle projects which make use of JNI code. I was met by a stony wall of silence at that time... Anyway, a month later, and after quite a lot of thinking, hacking and tweaking, I've come up with a solution I'm reasonably happy with. I've written it all up at http://docs.codehaus.org/display/MAVENUSER/Projects+With+JNI, in the hope that it may be useful to others. Feedback is welcome, although bear in mind that I've spent long enough on this now that I may be reluctant to do much more work :). Cheers, Rich -- Richard van der Hoff [EMAIL PROTECTED] Telephony Gateways Project Manager Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
Hi Richard, The native-maven-plugin JNI example shows that you can have your native project depends on a java jar project that has java jni interfaces. This way you can use native plugin to generate jni native include files before building the native artifacts. This is the same behavior in maven1. Not sure why it is a blocking to you since it works -Dan On 6/14/06, Richard van der Hoff [EMAIL PROTECTED] wrote: A while ago, I posted here asking for help with using Maven to handle projects which make use of JNI code. I was met by a stony wall of silence at that time... Anyway, a month later, and after quite a lot of thinking, hacking and tweaking, I've come up with a solution I'm reasonably happy with. I've written it all up at http://docs.codehaus.org/display/MAVENUSER/Projects+With+JNI, in the hope that it may be useful to others. Feedback is welcome, although bear in mind that I've spent long enough on this now that I may be reluctant to do much more work :). Cheers, Rich -- Richard van der Hoff [EMAIL PROTECTED] Telephony Gateways Project Manager Tel: +44 (0) 845 666 7778 http://www.mxtelecom.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: suggestions for handling native libraries
You seem to be on the right track. We'd certainly welcome contributions
along these lines, and there are a few people working on this elsewhere
already as you've highlighted.
Have you seen
http://docs.codehaus.org/display/MAVEN/Support+for+other+languages?
- Brett
Richard van der Hoff wrote:
Hi all,
Apologies in advance for the length of this mail. As you'll see, I've
been thinking about this for a while, but I want to get it right first
time rather than end up supporting legacy builds in the future. I'm
looking for suggestions on how best to handle native libraries. We have
quite a few projects here which have JNI code, so I'm after a neat,
reusable solution.
I'm keen to avoid the structure proposed at
http://svn.codehaus.org/trunk/mojo/maven-native/native-maven-plugin/src/it/jni/?root=mojo
which has quite a nested format, which seems to me to (a) cause
unnecessary clutter and (b) limit sharing between projects. Better would
be just one 'native' subproject (alongside a 'java' subproject). I'm not
going to rule out the whole native/{Linux,win32,etc} system - it just
isn't clear to me at the moment that it helps much.
Anyway, I face a whole array of problems :/
The first is how an application finds its native libraries.
System.loadLibrary() loads its libraries from the System property
java.library.path. The easiest way of achieving this seems to be to
use an assembly to build an archive containing the libraries, so that
they can just be loaded from the correct directory. Bit of a pain to
have to build and extract an assembly on each build cycle, but there you
go.
However, there is also a problem with what the library file is named.
Having the version number in the filename would mean that the java side
needs to know about the version of the native library (rather than just
transparently loading whatever happens to be there), so I'd rather avoid
that. Furthermore, different platforms have different requirements for
the naming of their jni libs - for instance, for a library foo, Linux
requires libfoo.so, win32 requires foo.dll, etc - all of which is
somewhat incompatible with the naming conventions of maven artifacts.
This brings me to my next thought, which is that the maven artifact
probably ought to be a zip file (or similar) containing the
correctly-named library. This is good because it means my artifacts have
the same extension no matter what platform they were built for; I can
distinguish between platforms using the classifier; and I can be sure
that the name of the library itself is just right for the given platform.
On the other hand, it means I now need to set up my assembly such that
the zip files are unpacked. Ideally, I'd like for all such artifacts to
be unpacked automatically, but there doesn't seem to be any way of doing
so without configuring it in each assembly.xml. It's a property of the
artifact, rather than the assembly, so it would be nice to somehow
derive this from an ArtifactHandler rather than in the assembly
descriptor. I can probably fudge this with a shared assembly descriptor,
however.
As an alternative approach, I considered storing the artifact in the
repository with the right prefix and suffix, and maybe letting the java
side know the right version with some sort of properties magic - which
might make the building of assemblies simpler, and would even allow you
to do without an assembly if you set up your java.library.path to
include bits of the local repository. However, that would seem to
require that the DefaultRepositoryLayout allows the ArtifactHandler to
specify a prefix as well as an extension, so that's a bit of a
fundamental change.
I'm assuming either solution is going to involve me creating my own mojo
to actually create the relevant artifact in the first place - but I'm
going to think harder about that once I've decided what I want the
repository to contain!
So, any thoughts, comments, criticism or praise on my line of thinking
above would be very much appreciated.
Cheers
Rich
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
