Re: Problem loading jcc from java : undefined symbol: PyExc_IOError

2011-02-15 Thread Roman Chyla
In the end, I compiled a new python with the necessary modules, and
that works just fine.
But it was an interesting experience. Thank you Andi, your help is always great.

Cheers,

  roman

On Tue, Feb 15, 2011 at 9:22 AM, Roman Chyla  wrote:
> On Tue, Feb 15, 2011 at 4:22 AM, Andi Vajda  wrote:
>>
>> On Tue, 15 Feb 2011, Roman Chyla wrote:
>>
>>> from:
>>> http://realmike.org/blog/2010/07/18/python-extensions-in-cpp-using-swig/
>>>
>>> Q. ?Fatal Python error: Interpreter not initialized (version mismatch?)?
>>>
>>> A. This error occurs when the version of the Python interpreter for
>>> which the extension module has been built is different from the
>>> version of the interpreter that attempts to import the module.
>>>
>>> Is there a way to find out which python interpreter version is inside
>>> JCC? Also, Is it somehow possible that the java process that load jcc
>>> library will be picking the default python (2.4) instead of the python
>>> (2.5)? PATH is set to python2.5.
>>
>> There is no Python interpreter inside jcc. It's dynamically linked.
>> To know which version of the shared library is looked for and expected, use
>> the 'ldd' utility against the various shared libraries involved to tell you.
>> That version is selected at build time, when you run 'python setup.py ...'
>> That version of python determines the version of libpython.so used.
>
> This will be probably the problem (as you said before), the libjcc.so
> shows no python -
>
> bash-3.2$ ldd build/lib.linux-x86_64-2.5/libjcc.so
>        linux-vdso.so.1 =>  (0x7fff7affc000)
>        /$LIB/snoopy.so => /lib64/snoopy.so (0x2b8ed0e74000)
>        libjava.so => 
> /afs/cern.ch/user/r/rchyla/public/jdk1.6.0_18/jre/lib/amd64/libjava.so
> (0x2b8ed1076000)
>        libjvm.so => 
> /afs/cern.ch/user/r/rchyla/public/jdk1.6.0_18/jre/lib/amd64/server/libjvm.so
> (0x2b8ed11a5000)
>        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2b8ed1c3f000)
>        libm.so.6 => /lib64/libm.so.6 (0x2b8ed1f3f000)
>        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2b8ed21c2000)
>        libpthread.so.0 => /lib64/libpthread.so.0 (0x2b8ed23cf000)
>        libc.so.6 => /lib64/libc.so.6 (0x2b8ed25eb000)
>        libdl.so.2 => /lib64/libdl.so.2 (0x2b8ed2943000)
>        libverify.so =>
> /afs/cern.ch/user/r/rchyla/public/jdk1.6.0_18/jre/lib/amd64/libverify.so
> (0x2b8ed2b47000)
>        libnsl.so.1 => /lib64/libnsl.so.1 (0x2b8ed2c57000)
>        /lib64/ld-linux-x86-64.so.2 (0x2b8ed08c9000)
>
> And I think, the python2.4 (the default  on the system) is being
> loaded -- but how to force loading of python2.5 (if that was possible
> at all) I don't know. Compilation is definitely done with -lpython2.5
>
> Cheers,
>
>  roman
>
>>
>> Andi..
>>
>>>
>>> Cheers,
>>>
>>>  roman
>>>
>>>
>>> On Tue, Feb 15, 2011 at 2:40 AM, Roman Chyla 
>>> wrote:

 On Tue, Feb 15, 2011 at 1:32 AM, Andi Vajda  wrote:
>
> On Tue, 15 Feb 2011, Roman Chyla wrote:
>
>> The python embedded in Java works really well on MacOsX and also
>> Ubuntu. But I am trying hard to make it work also on Scientific Linux
>> (SLC5) with *statically* built Python. The python is a build from
>> ActiveState.
>
> You mean you're going to try to dynamically load libpython.a into a JVM
> ?
> I have no idea if this can work at all.

 I am very ignorant as far as the difference between statically and
 dynamically linked libraries go - I just wanted to use JCC wrapped
 code with this particular statically linked python

 I got little bit further, but just little:

 after I changed -Xlinker --export-dynamic into "-Xlinker
 -export-dynamic" (and installed python into /opt...) I am getting a
 different error:

 SEVERE: org.apache.jcc.PythonException: No module named
 solrpie.java_bridge
 null
        at org.apache.jcc.PythonVM.instantiate(Native Method)
        at rca.python.jni.PythonVMBridge.start(Unknown Source)
        at rca.python.jni.PythonVMBridge.start(Unknown Source)
        at rca.python.jni.PythonVMBridge.start(Unknown Source)
        at rca.python.jni.SolrpieVM.getBridge(Unknown Source)


 My understanding is that the previous error has gone (and the python
 module "time" is loaded), because if I set PYTHONPATH incorrectly, I
 get:
 This message is IMHO coming from Python

 But when I correct the PYTHONPATH, I am getting only this:

 [java] Fatal Python error: Interpreter not initialized (version
 mismatch?)
 [java] Java Result: 134



>
> If my understanding of static builds is correct, I'd imagine the only
> way
> for this to work would be to statically compile the JVM (hotspot) and
> python
> together.

 oooups, that is way over my head

>
> But why all this ?

 Because on the grid, we already had a statically linked python and 

Problem loading jcc from java : undefined symbol: PyExc_IOError

2011-02-14 Thread Roman Chyla
Hello Andi, all,

The python embedded in Java works really well on MacOsX and also
Ubuntu. But I am trying hard to make it work also on Scientific Linux
(SLC5) with *statically* built Python. The python is a build from
ActiveState.

So far, I managed to build all the needed extensions (jcc, lucene,
solr) and I can run them in python, but when I try to start the java
app and use python, I get:

SEVERE: org.apache.jcc.PythonException:
/afs/cern.ch/user/r/rchyla/public/ActivePython-2.5.5.7-linux-x86_64/INSTALLDIR/lib/python2.5/lib-dynload/time.so:
undefined symbol: PyExc_IOError


I understand, that the missing symbol PyExc_IOError is in the static
python library:

bash-3.2$ nm 
/afs/cern.ch/user/r/rchyla/public/ActivePython-2.5.5.7-linux-x86_64/INSTALLDIR/lib/python2.5/config/libpython2.5.a
| grep IOError
4120 D PyExc_IOError
4140 d _PyExc_IOError
 U PyExc_IOError
 U PyExc_IOError
 U PyExc_IOError
 U PyExc_IOError
 U PyExc_IOError
 U PyExc_IOError
 U PyExc_IOError

So when building JCC, I build with these arguments:

lflags  +  ['-lpython%s.%s' %(sys.version_info[0:2]),
'-L',
'/afs/cern.ch/user/r/rchyla/public/ActivePython-2.5.5.7-linux-x86_64/INSTALLDIR/lib/python2.5/config',
'-rdynamic',
'-Wl,--export-dynamic',
'-Xlinker',
'--export-dynamic']

I just found instructions at:
http://stackoverflow.com/questions/4223312/python-interpreter-embedded-in-the-application-fails-to-load-native-modules
I don't really understand g++, but the symbol is there after the compilation

bash-3.2$ nm 
/afs/cern.ch/user/r/rchyla/public/ActivePython-2.5.5.7-linux-x86_64/INSTALLDIR/lib/python2.5/site-packages/JCC-2.7-py2.5-linux-x86_64.egg/libjcc.so
| grep IOError
00352240 D PyExc_IOError
00352260 d _PyExc_IOError

And when starting java, I do
"-Djava.library.path=/afs/cern.ch/user/r/rchyla/public/ActivePython-2.5.5.7-linux-x86_64/INSTALLDIR/lib/python2.5/site-packages/JCC-2.7-py2.5-linux-x86_64.egg"

The code works find on mac (python 2.6) and ubuntu (python2.6), but
not this statically linked python2.5 - would you know what I can try?

Thanks.


  roman


PS: I tried several compilations, but I was usually re-compiling JCC
without building lucene etc again, I hope that is not the problem.