Hi Andy,
This is much more than I could have hoped! Just yesterday, I was
looking for ways how to embed Python VM in Jetty, as that would be
more natural, but found only jepp.sourceforge.net and off-putting was
the necessity to compile it against the newly built python. I could
not want it from the guys who may need my extension. And I realize
only now, that embedding Python in Java is even documented on the
website, but honestly i would not know how to do it without your
detailed examples.
Now to the questions, I apologize, some of them or all must seem very
stupid to you
- pylucene is used on many platforms and with jcc always worked as
expected (i love it!), but is it as reliable in the opposite
direction? The PythonVM.java loads "jcc" library, so I wonder if in
principle there is any difference in the directionality - but I am not
sure. To rephrase my convoluted question: would you expect this
wrapping be as reliable as wrapping java inside python is now?
- in the past, i built jcc libraries on one host and distributed them
on various machines. As long the family OS and the python main version
were the same, it worked on Win/Lin/Mac just fine. As far as I can
tell, this does not change, or will it be dependent on the python
against which the egg was built?
- now a little tricky issue; when I wrap jetty inside python, I hoped
to build it in a shared mode with lucene to be able to do some
low-level lucene indexing tasks from inside Python. If I do the
opposite and wrap Python VM in Java, I would still like to access the
lucene (which is possible, as I see well from your examples) But on
the python side, you are calling initVM() - will the initVM() call
create a new Java VM or will it access the parent Java VM which
started it?
- you say that threads are not managed by the Python VM, does that
mean there is no Python GIL?
- I don't really know what is exactly in the python thread local
storage, could that somehow negatively affect the Python process if
acquireThreadState/releaseThreadState are not called?
Thank you.
Cheers,
roman
On Tue, Jan 11, 2011 at 8:13 PM, Andi Vajda <va...@apache.org> wrote:
>
> Hi Roman,
>
> On Tue, 11 Jan 2011, Roman Chyla wrote:
>
>> I have recently wrapped solr inside jetty with JCC (we need to access
>> very big result sets quickly, via JNI, but also keep solr running as
>> normal) and was wondering what strategies do you guys use to speak
>> *from inside* Java towards the Python end.
>>
>> So far, I was able to think about these:
>>
>> - raise exceptions in java and catch in python (I think I have seen
>> this in some posts from Bill Jansen)
>> - communicate via sockets
>> - wait passively - call some java method and wait for its return
>> - monitor actively - in python check in loop some java object
>>
>> Is there something else?
>
> I'm not sure I completely understand your questions but if what you're
> asking is how to run Python code from inside a Java servlet container, that
> I've done with Tomcat and Lucene.
>
> Basically, instead of embedding a JVM inside a Python VM - as is done for
> PyLucene - you do the opposite, you embed a Python VM inside a JVM.
>
> For that purpose, see the org.apache.jcc.PythonVM class available in JCC's
> java tree. This class must be instantiated from the main thread at Java
> servlet engine startup time. In Tomcat, I patched some startup code, in
> BootStrap.java (see patches below) for this purpose.
>
> Then, to make some Python code accessible from Java, use the usual way of
> writing "extensions", the so-called JCC in reverse trick. Define a Java
> class
> with some native methods implemented in Python; define a Python class that
> "extends" it; build the Java class into a JAR; include it into a JCC-built
> egg; install the egg into Python's env (site-packages, PYTHONPATH,
> whatever);
> Then, write servlet code in Java that imports your Java class and calls it.
>
> As you can see, this sounds simple but the devil is in the details. Of
> course,
> bending Jetty for this may have different requirements but the code snippets
> below should give you a good idea about what's required.
>
> This approach has been in production running the freebase.com's search
> server
> for over two years now.
>
> If you have questions, of course, please ask.
> Good luck !
>
> Andi..
>
> ----------------------
> Patch to Bootstrap.java to use JCC's PythonVM (which initializes the
> embedded
> Python VM)
>
> --- apache-tomcat-6.0.29-src/java/org/apache/catalina/startup/Bootstrap.java
> 2010-07-19 06:02:32.000000000 -0700
> +++
> apache-tomcat-6.0.29-src/java/org/apache/catalina/startup/Bootstrap.java.patched
> 2010-08-04 08:49:05.000000000 -0700
> @@ -30,16 +30,18 @@
> import javax.management.MBeanServer;
> import javax.management.MBeanServerFactory;
> import javax.management.ObjectName;
>
> import org.apache.catalina.security.SecurityClassLoad;
> import org.apache.juli.logging.Log;
> import org.apache.juli.logging.LogFactory;
>
> +import org.apache.jcc.PythonVM;
> +
>
> /**
> * Boostrap loader for Catalina. This application constructs a class loader
> * for use in loading the Catalina internal classes (by accumulating all of
> the
> * JAR files found in the "server" directory under "catalina.home"), and
> * starts the regular execution of the container. The purpose of this
> * roundabout approach is to keep the Catalina internal classes (and any
> * other classes they depend on, such as an XML parser) out of the system
> @@ -398,22 +400,24 @@
> try {
> String command = "start";
> if (args.length > 0) {
> command = args[args.length - 1];
> }
>
> if (command.equals("startd")) {
> args[args.length - 1] = "start";
> + PythonVM.start("mql");
> daemon.load(args);
> daemon.start();
> } else if (command.equals("stopd")) {
> args[args.length - 1] = "stop";
> daemon.stop();
> } else if (command.equals("start")) {
> + PythonVM.start("mql");
> daemon.setAwait(true);
> daemon.load(args);
> daemon.start();
> } else if (command.equals("stop")) {
> daemon.stopServer(args);
> } else {
> log.warn("Bootstrap: command \"" + command + "\" does not
> exist.");
> }
>
> -----------------------------------------
> Define a Java class:
>
> package ....
>
> public class EMQL {
>
> private long pythonObject;
>
> public EMQL()
> {
> }
>
> public void pythonExtension(long pythonObject)
> {
> this.pythonObject = pythonObject;
> }
> public long pythonExtension()
> {
> return this.pythonObject;
> }
>
> public void finalize()
> throws Throwable
> {
> pythonDecRef();
> }
>
> public native void pythonDecRef();
>
> // the methods implemented in python
> public native String init(ME me);
> public native String emql_refresh(String tid, String type);
> public native String emql_status();
>
> etc .......... etc
>
> ------------------------------------
> The corresponding Python class
>
> import ......
>
> from jemql import initVM, CLASSPATH, EMQL
>
> initVM(CLASSPATH)
>
> class emql(EMQL):
>
> def __init__(self):
> super(emql, self).__init__()
>
> def init(self, me):
> ...........
> def emql_refresh(self, tid, type):
> ...........
> def emql_status(self):
> ...........
> return "some status"
>
> etc ...... etc
>
> ------------------------------------
> Makefile rules to build this via JCC (the jemql.egg file is just an empty
> target file for Makefile, it's not used for anything else):
>
> default: jemql.egg
>
> jemql.jar: java/org/blah/blah/EMQL.java
> mkdir -p classes
> javac -classpath $(CLASSPATH):$(MORE_CLASSPATH):$(etc..etc) -d
> classes $(JAVAC_FLAGS) $<
> jar -cvf $@ -C classes .
>
> jemql.egg: jemql.jar $(JMQL_JAR) emql.py
> $(JCC) --version 1.0 --jar $< \
> --classpath $(CLASSPATH):$(JME_JAR):$(JMQL_JAR) \
> org.blah.blah.me.ME \
> --package java.lang \
> --python jemql --build $(DBG_FLAGS) \
> --install \
> --module emql
> touch $@
> ------------------------------------
> Patch to Tomcat's build.xml ANT script to add JCC's classes (like PythonVM)
> to
> the build classpath.
>
> --- apache-tomcat-6.0.29-src/build.xml 2010-07-19 06:02:31.000000000 -0700
> +++ apache-tomcat-6.0.29-src/build.xml.patched 2010-08-04
> 09:30:24.000000000 -0700
> @@ -95,16 +95,17 @@
> <property name="jasper-jdt.jar"
> value="${jasper-jdt.home}/jasper-jdt.jar"/>
> <available property="tomcat-dbcp.present" file="${tomcat-dbcp.jar}" />
> <available property="jdk16.present" classname="javax.sql.StatementEvent"
> />
>
> <!-- Classpath -->
> <path id="tomcat.classpath">
> <pathelement location="${ant.jar}"/>
> <pathelement location="${jdt.jar}"/>
> + <pathelement location="${jcc.egg}/jcc/classes"/>
> </path>
>
> <!-- Version info filter set -->
> <tstamp>
> <format property="TODAY" pattern="MMM d yyyy" locale="en"/>
> <format property="TSTAMP" pattern="hh:mm:ss"/>
> </tstamp>
> <filterset id="version.filters">
> @@ -148,16 +149,25 @@
> excludes="**/CVS/**,**/.svn/**"
> encoding="ISO-8859-1">
> <!-- Comment this in to show unchecked warnings:
> <compilerarg value="-Xlint:unchecked"/>
> -->
> <classpath refid="tomcat.classpath" />
> <exclude name="org/apache/naming/factory/webservices/**" />
> </javac>
> + <javac srcdir="${extras.path}" destdir="${tomcat.classes}"
> + debug="${compile.debug}"
> + deprecation="${compile.deprecation}"
> + source="${compile.source}"
> + optimize="${compile.optimize}"
> + excludes="**/CVS/**,**/.svn/**">
> +<!-- Comment this in to show unchecked warnings: <compilerarg
> value="-Xlint:unchecked"/> -->
> + <classpath refid="tomcat.classpath" />
> + </javac>
> <!-- Copy static resource files -->
> <copy todir="${tomcat.classes}" encoding="ISO-8859-1">
> <filterset refid="version.filters"/>
> <fileset dir="java">
> <include name="**/*.properties"/>
> <include name="**/*.dtd"/>
> <include name="**/*.tasks"/>
> <include name="**/*.xsd"/>
>
> -----------------------------------------------
> Patch to catalina.sh, the Tomcat startup script to add JCC to LIBPATH and
> CLASSPATH
>
> --- apache-tomcat-6.0.29-src/output/build/bin/catalina.sh 2010-08-04
> 09:57:27.000000000 -0700
> +++ apache-tomcat-6.0.29-src/output/build/bin/catalina.sh.patched
> 2010-08-04 09:57:47.000000000 -0700
> @@ -162,16 +162,30 @@
> exit 1
> fi
> fi
>
> if [ -z "$CATALINA_BASE" ] ; then
> CATALINA_BASE="$CATALINA_HOME"
> fi
>
> +if [ -n "$JCC_EGG" ]; then
> + CLASSPATH="$CLASSPATH":"$JCC_EGG"/jcc/classes
> + JAVA_LIB_PATH=$JCC_EGG
> +fi
> +if [ -n "$TOMCAT_APR_LIB_PATH" ]; then
> + JAVA_LIB_PATH=$JAVA_LIB_PATH:$TOMCAT_APR_LIB_PATH
> +fi
> +if [ -n "$JAVA_LIB_PATH" ]; then
> + JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$JAVA_LIB_PATH"
> +fi
> +if [ -n "EXTRA_CLASSPATH" ]; then
> + CLASSPATH="$CLASSPATH":"$EXTRA_CLASSPATH"
> +fi
> +
> # Add tomcat-juli.jar and bootstrap.jar to classpath
> # tomcat-juli.jar can be over-ridden per instance
> if [ ! -z "$CLASSPATH" ] ; then
> CLASSPATH="$CLASSPATH":
> fi
> if [ "$CATALINA_BASE" != "$CATALINA_HOME" ] && [ -r
> "$CATALINA_BASE/bin/tomcat-juli.jar" ] ; then
>
> CLASSPATH="$CLASSPATH""$CATALINA_BASE"/bin/tomcat-juli.jar:"$CATALINA_HOME"/bin/bootstrap.jar
> else
>
> These EGG paths are long, complicated and OS-specific, the trick below
> generates them programmatically (from inside a Makefile):
>
> JCC_EGG:=$(shell $(PYTHON) -c "import os, jcc; print
> os.path.dirname(os.path.dirname(jcc.__file__))")
> JEMQL_EGG:=$(shell $(PYTHON) -c "import os, jemql; print
> os.path.dirname(os.path.dirname(jemql.__file__))")
>
> Then, the CLASSPATH addition during _build_ time:
> CLASSPATH = $(CLASSPATH):$(JEMQL_EGG)/jemql/jemql.jar
> and so on...
> At runtime, JCC takes care of adding your eggs to the startup CLASSPATH.
>
> ----------------------------------------------
> Last but not least, if you use Python's thread local storage in your
> threads, Python threads when embedded inside a JVM are 'dummy', that is,
> while they're
> backed by the actual Java thread (a pthread), the Python VM is not managing
> them and a thread state object is created each and every time a Python
> thread
> is entered and released when exited back to the JVM. This has two problems:
> 1. it's a bit wasteful
> 2. python thread local storage gets lost
>
> The Java class below works this around by incrementing the refcount that
> controls this:
>
> package org.apache.catalina.core;
>
> import org.apache.jcc.PythonVM;
>
> public class TerminatingThread extends Thread {
> protected Runnable runnable;
>
> public TerminatingThread(ThreadGroup group, Runnable runnable, String
> name)
> {
> super(group, name);
> this.runnable = runnable;
> }
>
> public void run()
> {
> PythonVM vm = PythonVM.get();
>
> try {
> vm.acquireThreadState();
> runnable.run();
> } finally {
> vm.releaseThreadState();
> }
> }
> }
>
> Then, there is some trickery to get Tomcat to use this class for its threads
> instead of the default one:
>
> ---
> apache-tomcat-6.0.29-src/java/org/apache/catalina/core/StandardThreadExecutor.java
> 2010-07-19 06:02:32.000000000 -0700
> +++
> apache-tomcat-6.0.29-src/java/org/apache/catalina/core/StandardThreadExecutor.java.patched
> 2010-08-04 08:56:02.000000000 -0700
> @@ -44,17 +44,17 @@
> protected int minSpareThreads = 25;
>
> protected int maxIdleTime = 60000;
>
> protected ThreadPoolExecutor executor = null;
>
> protected String name;
>
> - private LifecycleSupport lifecycle = new LifecycleSupport(this);
> + protected LifecycleSupport lifecycle = new LifecycleSupport(this);
> // ---------------------------------------------- Constructors
> public StandardThreadExecutor() {
> //empty constructor for the digester
> }
>
>
>
> // ---------------------------------------------- Public Methods
>
>
> In Tomcat's server.xml, use this executor (and code below for it)
> <Executor name="relThreadPool"
> className="org.apache.catalina.core.TerminatingThreadExecutor"
> namePrefix="rel-exec-"
> maxIdleTime="3600000"
> minSpareThreads="2"
> maxThreads="2" />
>
>
> package org.apache.catalina.core;
>
> import java.util.concurrent.ThreadPoolExecutor;
> import java.util.concurrent.TimeUnit;
> import org.apache.catalina.LifecycleException;
>
>
> public class TerminatingThreadExecutor extends StandardThreadExecutor {
>
> public void start()
> throws LifecycleException
> {
> lifecycle.fireLifecycleEvent(BEFORE_START_EVENT, null);
>
> TaskQueue taskqueue = new TaskQueue();
> TaskThreadFactory tf = new TerminatingTaskThreadFactory(namePrefix);
>
> lifecycle.fireLifecycleEvent(START_EVENT, null);
> executor = new ThreadPoolExecutor(getMinSpareThreads(),
> getMaxThreads(),
> maxIdleTime, TimeUnit.MILLISECONDS,
> taskqueue, tf);
> taskqueue.setParent(executor);
> lifecycle.fireLifecycleEvent(AFTER_START_EVENT, null);
> }
>
> protected class TerminatingTaskThreadFactory
> extends StandardThreadExecutor.TaskThreadFactory {
>
> protected TerminatingTaskThreadFactory(String namePrefix)
> {
> super(namePrefix);
> }
>
> public Thread newThread(Runnable runnable)
> {
> Thread t = new TerminatingThread(group, runnable, namePrefix +
> threadNumber.getAndIncrement());
>
> t.setDaemon(daemon);
> t.setPriority(getThreadPriority());
>
> return t;
> }
> }
> }
>
