Hey Michael, thanks for the investigative work you put into this, and sorry for not replying earlier.
I was not aware of the thread limitations of JPype, that's quite dissapointing.. Overall, it feels like the JPype solution is not a manageable one in the long term. The embedded use case is cool, because it's so freaking convenient, but using JPype to embed the JVM is painful. There are a few other options for JVM->python integration that are starting to look interesting, perhaps we can look around a little bit and see what is available. It would be especially cool if we can use something that works across CPython, Jython as well as PyPy. I know Tobias was thinking about something that would make that possible. In the short to medium term though, I'd like to focus on two things (please note that this is *my opinion*, not a product roadmap): *Getting cypher and gremlin into the embedded python API* Eventually, I think we should deprecate the traversal API in the embedded client in favor of Cypher. Traversals need callbacks and shared state and so on between the two VMs, and its gonna hurt, I think, if we depend on that further down the road. Doing this will also make it simple to keep full 1-1 API compatibility between the embedded client and REST clients, making it easy for end users to switch between the two. *Making the REST API fast enough to make that use case more appealing.* JPype is really just a cheap way of implementing a form of remoting API. It's currently worth the pain, since it is a ton faster than our REST API, but it's a band-aid, not a proper fix. We should focus on speeding up the REST API. WDYT? /Jake On Tue, Nov 1, 2011 at 5:00 PM, Michael Rene Armida <m...@marmida.com> wrote: > I solved my problem by ensuring that I don't 'import neo4j' until I'm in a > dedicated thread. So long as all neo4j interaction is handled within that > thread, I don't crash. > > The Python neo4j bindings use JPype; I noticed this note from the JPype > docs (http://jpype.sourceforge.net/doc/user-guide/userguide.html#threading > ): > > "For the most part, python threads based on OS level threads (i.e posix > threads), will work without problem. The only thing to remember is to > call jpype.attachThreadToJVM() in the thread body to make the JVM usable > from that thread. For threads that you do not start yourself, you can > call isThreadAttachedToJVM() to check." > > I couldn't find any evidence for this, but I think that some of the Java > code in the neo4j bindings may call attachThreadToJVM at import time. > > Unfortunately, this means that even though neo4j is thread-safe, it must be > constrained to a single thread when used from Python. But that's not too > disappointing, considering. > > On Sat, Oct 29, 2011 at 4:25 PM, Michael Rene Armida <m...@marmida.com> > wrote: > > > I am attempting to use Paste to serve a small web app that uses neo4j via > > the Python bindings. I get a segfault whenever I try to create a new > > database via 'neo4j.GraphDatabase'. I have narrowed this down to the > > following minimal sample: > > > > http://pastebin.com/MkNRvLR6 > > > > The top of the stack trace looks like this: > > > > Stack: [0xb42e7000,0xb4ae8000], sp=0xb4ae44f0, free space=8181k > > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > > code) > > C [_jpype.so+0x26497] JPJavaEnv::NewObjectA(_jclass*, _jmethodID*, > > jvalue*)+0x37 > > C [_jpype.so+0x3c0e8] JPMethodOverload::invokeConstructor(_jclass*, > > std::vector<HostRef*, std::allocator<HostRef*> >&)+0x178 > > C [_jpype.so+0x3a417] JPMethod::invokeConstructor(std::vector<HostRef*, > > std::allocator<HostRef*> >&)+0x47 > > C [_jpype.so+0x1beba] JPClass::newInstance(std::vector<HostRef*, > > std::allocator<HostRef*> >&)+0x2a > > C [_jpype.so+0x67b9c] PyJPClass::newClassInstance(_object*, > > _object*)+0xfc > > C [python+0x96822] PyEval_EvalFrameEx+0x4332 > > C [python+0x991e7] PyEval_EvalCodeEx+0x127 > > > > > > I believe that is JPype trying to create an EmbeddedGraphDatabase via > > neo4j.GraphDatabase() in python. > > > > I noticed this previous thread: > > > http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-CPython-Pylons-and-threading-td942435.html > > > > That looks like it might be related, except this problem occurs whenever > I > > try to create a database, not on shutdown. The issue definitely has > > something to do with using neo4j under Paste; here's a Python shell > session > > which shows neo4j working: > > > > >>> import webob > > >>> import app > > >>> debug_app = app.DebugApp() > > >>> response = debug_app(webob.Request.blank('/')) > > >>> response.body > > 'it worked' > > > > Relevant version info: > > - ubuntu 11.04 (same crash under 32- and 64-bit systems) > > - Python 2.7.1-0ubuntu5 > > - neo4j_embedded-1.5.b2 > > - jpype 0.5.4.1-3 > > - paste 1.7.5.1 > > - OpenJDK 6b22-1.10.2 (same crash under the Sun JRE) > > > > > > Thanks for your time and any clues you might have. > > > _______________________________________________ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
