Preforking python process pool server
-------------------------------------

                 Key: THRIFT-1083
                 URL: https://issues.apache.org/jira/browse/THRIFT-1083
             Project: Thrift
          Issue Type: New Feature
          Components: Python - Library
    Affects Versions: 0.6
         Environment: linux. haven't tested on windows
            Reporter: Dvir Volk
            Priority: Minor


This patch adds a new server to the python library: TProcessPoolServer, which 
is a preforking python server.

this server is sort of a mix between TForkingServer and TThreadPoolServer: It 
has a pool of workers that are preforked subprocesses.
This approach allows the user to avoid the python GIL single processor limit in 
threading applications, and use a high performance server, and at the same time 
it avoids the need to fork a child process for each connection, as is the case 
with the forking server.

I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a 
quad Corei7 CPU, and about the same amount faster than TForkingServer if you 
are not using persistent connections and forking a child for each request.

The patch also updates the python unit tests to include tests for this server.

Notes:
1. Of course this server has the limitations of forking regarding shared state 
and memory copying. 
2. You should NOT kill the server with kill -9 - as this will not allow the 
parent process to terminate its children, resulting in orphaned processes that 
keep your socket open. either run stop() in your app or kill the process with 
SIGINT. It will respond to ctrl+C however
3. use setNumWorkers(n) before starting the server to determine how many 
processes you want to spawn. the default is 10.



-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Reply via email to