I've been researching/playing with Python web technologies all week long (my head is still spinning). My most current incarnation of my Searcher application: - wrote a custom WSGI request handler (similar to Colubrid) - used the above to write a WSGI Searcher app that outputs XML - wrote a script that launches the Searcher app using paste.httpserver (pythonpaste.org) I chose paste because I want to be able to run the searcher app as a stand-alone web server speaking http, or as an app server speaking SCGI behind lighttpd. Paste comes with both paste.httpserver and paste.util.scgiserver, so it should be easy. Paste segfaulted at first, until I remembered that all threads must be PythonThread, so I hacked the httpserver.py file (just import PyLucene and modify line 513) to use PythonThread, and now it works, except for one problem... I want to be able to run the app as a daemon, where it detaches from the terminal and returns control back to bash. I found this recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278731 I refactored the code a bit into a function called daemonize() which I put in a utility module, and then added a command line arg to my searcher script to invoke it as a daemon. I also have a test script that connects to the Searcher app and runs a long set of queries against it, scoring the quality of the output. It <http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/278731> works fine when run the Searcher a normal process, but when I run the Searcher as a daemon, it freezes after about a dozen queries, in the middle of the queryparser.parse() call. It never gets past that point. I wrote a SIGTERM handler for shutting down the daemon, and it works fine, until the Searcher freezes, and then it no longer responds to the SIGTERM. Not sure where to go from here. I see that Flup talks FastCGI and SCGI. Does it talk HTTP as well? -ofer
_____ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ninneman, TJ Sent: Friday, March 23, 2007 12:33 PM To: [email protected] Subject: [pylucene-dev] Need to build a high-load searcher I didn't actually have much luck with running PyLucene within a FastAGI instance. I used the combination of lighttpd + Flup and it worked just fine until I placed some load on it. Whether running in prefork or threaded mode, I would constantly receive "Collecting from unknown thread" errors (which really surprised me in prefork mode). I then thought that I could modify Flup to use the PythonThread class. Unfortunately, Flup does not use the threading module (of which PythonThread is a reimplementation of) but instead uses "start_new_thread()" from the thread module to fire it's threads. Now, I'm sure that a thread-savvy pythonista could make this work but I just didn't have the time to do it myself. Instead, I utilized the built-in SimpleXMLRPCServer and a sub-classed ThreadingMixIn: import SocketServer from SimpleXMLRPCServer import SimpleXMLRPCServer class MyLuceneHandler: def search(self, index, query): # Do my search pass class MyThreading(SocketServer.ThreadingMixIn): def process_request(self, request, client_address): """Start a new PyLucene safe thread to handle the request.""" from PyLucene import PythonThread t = PythonThread(target = self.process_request_thread, args = (request, client_address)) if self.daemon_threads: t.setDaemon (1) t.start() class ThreadingXMLRPC(MyThreading, SimpleXMLRPCServer): pass if __name__ == '__main__': server = ThreadingXMLRPC(('', int(sys.argv[1]))) server.register_instance(ProductsIndex()) server.serve_forever() It's really quite simple and hasn't let us down just yet. With that being said, if anybody has a working FastCGI method, I'm all ears!!! I think the benefits of using FastCGI from lighttpd are fantastic. TJ Ninneman
_______________________________________________ pylucene-dev mailing list [email protected] http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
