pylucene branch_3x fails to build
First, the patch jcc/jcc/patches/patch.43.0.6c11 doesn't quite work
for Python2.7.1. I have attached a modified patch.
More serious is the following error during make
(I'm using sun-jdk-1.6.0.23)
ant -f extensions.xml -Dlucene.dir=lucene-java-3.x
Buildfile: /Work1/Obj/Python/pylucene-build/extensions.xml
compile:
[mkdir] Created dir: /Work1/Obj/Python/pylucene-build/build/classes
[javac] /Work1/Obj/Python/pylucene-build/extensions.xml:19:
warning: 'includeantruntime' was not set, defaulting to
build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 29 source files to /Work1/Obj/Python/pylucene-
build/build/classes
[javac] /Work1/Obj/Python/pylucene-build/java/org/apache/pylucene/
search/PythonSimilarity.java:25:
org.apache.pylucene.search.PythonSimilarity is not abstract and does
not override abstract method computeNorm
(java.lang.String,org.apache.lucene.index.FieldInvertState) in
org.apache.lucene.search.Similarity
[javac] public class PythonSimilarity extends Similarity {
[javac]^
[javac] /Work1/Obj/Python/pylucene-build/java/org/apache/pylucene/
search/PythonSimilarity.java:70: lengthNorm(java.lang.String,int) in
org.apache.pylucene.search.PythonSimilarity cannot override lengthNorm
(java.lang.String,int) in org.apache.lucene.search.Similarity;
overridden method is final
[javac] public native float lengthNorm(String fieldName, int
numTokens);
[javac] ^
[javac] /Work1/Obj/Python/pylucene-build/java/org/apache/pylucene/
search/PythonSimilarityDelegator.java:49: lengthNorm
(java.lang.String,int) in
org.apache.pylucene.search.PythonSimilarityDelegator cannot override
lengthNorm(java.lang.String,int) in
org.apache.lucene.search.Similarity; overridden method is final
[javac] public native float lengthNorm(String fieldName, int
numTokens);
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] 3 errors
BUILD FAILED
/Work1/Obj/Python/pylucene-build/extensions.xml:19: Compile failed; see
the compiler error output for details.
Thanks for looking into it,
Helmut.
--
Helmut Jarausch
Lehrstuhl fuer Numerische Mathematik
RWTH - Aachen University
D 52056 Aachen, Germany
--- setuptools/extension.py.ORIG 2010-07-15 01:53:38.0 +0200
+++ setuptools/extension.py 2011-01-11 09:31:35.737668366 +0100
@@ -28,6 +28,11 @@
class Library(Extension):
Just like a regular Extension, but built as a library instead
+def __init__(self, *args, **kwds):
+self.force_shared = kwds.pop('force_shared', False)
+Extension.__init__(self, *args, **kwds)
+
+
import sys, distutils.core, distutils.extension
distutils.core.Extension = Extension
distutils.extension.Extension = Extension
--- setuptools/command/build_ext.py.ORIG 2011-01-11 09:30:11.0 +0100
+++ setuptools/command/build_ext.py 2011-01-11 09:33:33.55899 +0100
@@ -91,8 +91,12 @@
return filename
ext = self.ext_map[fullname]
if isinstance(ext,Library):
+if ext.force_shared and not use_stubs:
+ _libtype = 'shared'
+else:
+ _libtype = libtype
fn, ext = os.path.splitext(filename)
-return self.shlib_compiler.library_filename(fn,libtype)
+return self.shlib_compiler.library_filename(fn,_libtype)
elif use_stubs and ext._links_to_dynamic:
d,fn = os.path.split(filename)
return os.path.join(d,'dl-'+fn)
@@ -182,14 +186,22 @@
def build_extension(self, ext):
_compiler = self.compiler
try:
+force_shared = False
if isinstance(ext,Library):
self.compiler = self.shlib_compiler
+force_shared = ext.force_shared and not use_stubs
+if force_shared:
+self.compiler.link_shared_object = \
+sh_link_shared_object.__get__(self.compiler)
_build_ext.build_extension(self,ext)
if ext._needs_stub:
self.write_stub(
self.get_finalized_command('build_py').build_lib, ext
)
finally:
+if force_shared:
+self.compiler.link_shared_object = \
+link_shared_object.__get__(self.compiler)
self.compiler = _compiler
def links_to_dynamic(self, ext):
@@ -256,44 +268,41 @@
os.unlink(stub_file)
+def sh_link_shared_object(self, objects, output_libname, output_dir=None,
+libraries=None, library_dirs=None, runtime_library_dirs=None,
+export_symbols=None, debug=0, extra_preargs=None,
+extra_postargs=None, build_temp=None, target_lang=None
+): self.link(self.SHARED_LIBRARY, objects, output_libname,
+ output_dir, libraries, library_dirs, runtime_library_dirs,
+
Re: call python from java - what strategy do you use?
Hi Andy, This is much more than I could have hoped! Just yesterday, I was looking for ways how to embed Python VM in Jetty, as that would be more natural, but found only jepp.sourceforge.net and off-putting was the necessity to compile it against the newly built python. I could not want it from the guys who may need my extension. And I realize only now, that embedding Python in Java is even documented on the website, but honestly i would not know how to do it without your detailed examples. Now to the questions, I apologize, some of them or all must seem very stupid to you - pylucene is used on many platforms and with jcc always worked as expected (i love it!), but is it as reliable in the opposite direction? The PythonVM.java loads jcc library, so I wonder if in principle there is any difference in the directionality - but I am not sure. To rephrase my convoluted question: would you expect this wrapping be as reliable as wrapping java inside python is now? - in the past, i built jcc libraries on one host and distributed them on various machines. As long the family OS and the python main version were the same, it worked on Win/Lin/Mac just fine. As far as I can tell, this does not change, or will it be dependent on the python against which the egg was built? - now a little tricky issue; when I wrap jetty inside python, I hoped to build it in a shared mode with lucene to be able to do some low-level lucene indexing tasks from inside Python. If I do the opposite and wrap Python VM in Java, I would still like to access the lucene (which is possible, as I see well from your examples) But on the python side, you are calling initVM() - will the initVM() call create a new Java VM or will it access the parent Java VM which started it? - you say that threads are not managed by the Python VM, does that mean there is no Python GIL? - I don't really know what is exactly in the python thread local storage, could that somehow negatively affect the Python process if acquireThreadState/releaseThreadState are not called? Thank you. Cheers, roman On Tue, Jan 11, 2011 at 8:13 PM, Andi Vajda va...@apache.org wrote: Hi Roman, On Tue, 11 Jan 2011, Roman Chyla wrote: I have recently wrapped solr inside jetty with JCC (we need to access very big result sets quickly, via JNI, but also keep solr running as normal) and was wondering what strategies do you guys use to speak *from inside* Java towards the Python end. So far, I was able to think about these: - raise exceptions in java and catch in python (I think I have seen this in some posts from Bill Jansen) - communicate via sockets - wait passively - call some java method and wait for its return - monitor actively - in python check in loop some java object Is there something else? I'm not sure I completely understand your questions but if what you're asking is how to run Python code from inside a Java servlet container, that I've done with Tomcat and Lucene. Basically, instead of embedding a JVM inside a Python VM - as is done for PyLucene - you do the opposite, you embed a Python VM inside a JVM. For that purpose, see the org.apache.jcc.PythonVM class available in JCC's java tree. This class must be instantiated from the main thread at Java servlet engine startup time. In Tomcat, I patched some startup code, in BootStrap.java (see patches below) for this purpose. Then, to make some Python code accessible from Java, use the usual way of writing extensions, the so-called JCC in reverse trick. Define a Java class with some native methods implemented in Python; define a Python class that extends it; build the Java class into a JAR; include it into a JCC-built egg; install the egg into Python's env (site-packages, PYTHONPATH, whatever); Then, write servlet code in Java that imports your Java class and calls it. As you can see, this sounds simple but the devil is in the details. Of course, bending Jetty for this may have different requirements but the code snippets below should give you a good idea about what's required. This approach has been in production running the freebase.com's search server for over two years now. If you have questions, of course, please ask. Good luck ! Andi.. -- Patch to Bootstrap.java to use JCC's PythonVM (which initializes the embedded Python VM) --- apache-tomcat-6.0.29-src/java/org/apache/catalina/startup/Bootstrap.java 2010-07-19 06:02:32.0 -0700 +++ apache-tomcat-6.0.29-src/java/org/apache/catalina/startup/Bootstrap.java.patched 2010-08-04 08:49:05.0 -0700 @@ -30,16 +30,18 @@ import javax.management.MBeanServer; import javax.management.MBeanServerFactory; import javax.management.ObjectName; import org.apache.catalina.security.SecurityClassLoad; import org.apache.juli.logging.Log; import org.apache.juli.logging.LogFactory; +import org.apache.jcc.PythonVM; + /** * Boostrap loader for Catalina. This application
