Re: xmlrpc idea for getting around the GIL
On Wed, Dec 2, 2009 at 7:42 AM, sturlamolden wrote: > On 2 Des, 02:47, Patrick Stinson > wrote: > >> We don't need extension modules, and all we need to do is run some >> fairly basic scripts that make callbacks and use some sip-wrapped >> types. > > Sure, you use SIP but not extension modules... > > >> - Python is not suitable for real-time work. >> >> Not true. We have been running python callback code using >> PyObject_CallObject from within our audio thread for some time without >> a problem, and it's *extremely* fast. > > It seems you are confusing "real-time" with "real-fast". The fact that > something runs fast does not make it "real-time". > > Python is not suitable for real-time applications, nor are the OSes > commonly used to run Python. > Semantics aside, my point is that it runs well enough in our environment. If the audio is smooth, it is considered "working". > > >> We >> need just a ltle push to get our code to work at low latencies, >> and the only thing that is standing in our way is that all threads >> 9usually around 20 have to block on the Gil, and this causes small >> gaps in the sound at low latencies (around 5ms, or 64 sample period). >> >> ...almost perfect. > > Python is not programmed with real-time applications in mind: You have > no guarrantees on maximum time-lag when a thread is blocked on the > GIL. We don't need theoretical guarantees, because we've tried it and it works. That's the bottom line > > "Priority requests" (i.e. pre-emptive multitasking) was removed from > Antoine's "newgil" branch, but that is the kind of mechanism you would > need. Even with priority requests, Python would not be suitable for > real-time apps, as extension modules (e.g. C++ wrapped with SIP) can > hold the GIL forever. see above. > > You will also need an OS with a real-time scheduler and a real-time C > library, such as QNX or VxWorks. > > I find thea idea of a true real-time Python very interesting, but it > would take a completely reworked interpreter. > > > > > > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
On 2 Des, 02:47, Patrick Stinson wrote: > We don't need extension modules, and all we need to do is run some > fairly basic scripts that make callbacks and use some sip-wrapped > types. Sure, you use SIP but not extension modules... > - Python is not suitable for real-time work. > > Not true. We have been running python callback code using > PyObject_CallObject from within our audio thread for some time without > a problem, and it's *extremely* fast. It seems you are confusing "real-time" with "real-fast". The fact that something runs fast does not make it "real-time". Python is not suitable for real-time applications, nor are the OSes commonly used to run Python. > We > need just a ltle push to get our code to work at low latencies, > and the only thing that is standing in our way is that all threads > 9usually around 20 have to block on the Gil, and this causes small > gaps in the sound at low latencies (around 5ms, or 64 sample period). > > ...almost perfect. Python is not programmed with real-time applications in mind: You have no guarrantees on maximum time-lag when a thread is blocked on the GIL. "Priority requests" (i.e. pre-emptive multitasking) was removed from Antoine's "newgil" branch, but that is the kind of mechanism you would need. Even with priority requests, Python would not be suitable for real-time apps, as extension modules (e.g. C++ wrapped with SIP) can hold the GIL forever. You will also need an OS with a real-time scheduler and a real-time C library, such as QNX or VxWorks. I find thea idea of a true real-time Python very interesting, but it would take a completely reworked interpreter. -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
Patrick Stinson wrote: > Not true. We sell the industry leading sampler engine, and it has been > paying my salary for three years. It's high performance - every cycle > counts. Our sampled instruments is loaded as a plugin from third-party > applications and has been used to make movies you have seen. Our > customers love it and have never complained about problems with the > scripting engine. As an occasional bedroom producer I have to ask if you are talking about EastWest's Play? When was Python support added? I'm surprised I missed this at the time... -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
yes, using an rpc mechanism would insert a "blocking" call into a thread in which I am "not allowed to make a blocking call," but actual turn around times would be far better than forcing all threads to wait on the Gil. As it stands, blocking on a single thread lock *almost* works, and while we can claim this or that theoretical deficiency, the proof is in the pudding and the pudding almost works. In fact, it does work and has made us money, we just need one more once of performance at low latencies and I can put this way. Most machines that are running an audio sequencing host app are not running other apps, and we can generally assume that the host app is the only one using up the majority of resources on the local machine. So, as it turns out, we can assume quite a predictable and stable operating environment for our code. We don't need extension modules, and all we need to do is run some fairly basic scripts that make callbacks and use some sip-wrapped types. Py_CallObject is our only entry point into python execution from within the critical thread. SIDE NOTE: There are a lot of use cases that the educated python community is not familiar with that are really spectacularly interesting, and I think some of the myths surrounding python are incorrect. - Python is not suitable for real-time work. Not true. We have been running python callback code using PyObject_CallObject from within our audio thread for some time without a problem, and it's *extremely* fast. - Especially in a production environment. Not true. We sell the industry leading sampler engine, and it has been paying my salary for three years. It's high performance - every cycle counts. Our sampled instruments is loaded as a plugin from third-party applications and has been used to make movies you have seen. Our customers love it and have never complained about problems with the scripting engine. Audio work is interesting, because you can think and think and think about what will work and what wont, and at the end of the day all that matters is that the audio is smooth and the code doesn't crash. We need just a ltle push to get our code to work at low latencies, and the only thing that is standing in our way is that all threads 9usually around 20 have to block on the Gil, and this causes small gaps in the sound at low latencies (around 5ms, or 64 sample period). ...almost perfect. On Sun, Nov 29, 2009 at 8:57 AM, Aahz wrote: > In article <4b0b07a1$0$22159$9b622...@news.freenet.de>, > =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: >> >>In any case, I don't think you'll need a multi-process solution; a >>single-process multi-threading approach will do fine. Just create >>*another* thread, that runs at a low priority and is allowed to block. >>Run the Python interpreter in that thread (create multiple of these if >>you need them). Then, use some queuing producer-consumer communication >>between the high-priority thread and the low priority thread. Have the >>high priority threads put requests into the queue, and the low priority >>thread take them from the queue, dispatching them to Python. Make sure >>you use non-blocking synchronization on the queue, or else priority >>inheritance if you can get permission to do so. > > This is close to what I would recommend, but I would suggest that the > low-priority thread make calls out to an external Python process; that > way, you can have a Python worker pool of processes. Based on what I've > seen posted over the years, I generally recommend against instantiating > multiple interpreter instances (although I've seen some success stories, > I've seen more people butting their heads against the brick wall). > -- > Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ > > The best way to get information on Usenet is not to ask a question, but > to post the wrong information. > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
In article <4b0b07a1$0$22159$9b622...@news.freenet.de>, =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > >In any case, I don't think you'll need a multi-process solution; a >single-process multi-threading approach will do fine. Just create >*another* thread, that runs at a low priority and is allowed to block. >Run the Python interpreter in that thread (create multiple of these if >you need them). Then, use some queuing producer-consumer communication >between the high-priority thread and the low priority thread. Have the >high priority threads put requests into the queue, and the low priority >thread take them from the queue, dispatching them to Python. Make sure >you use non-blocking synchronization on the queue, or else priority >inheritance if you can get permission to do so. This is close to what I would recommend, but I would suggest that the low-priority thread make calls out to an external Python process; that way, you can have a Python worker pool of processes. Based on what I've seen posted over the years, I generally recommend against instantiating multiple interpreter instances (although I've seen some success stories, I've seen more people butting their heads against the brick wall). -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ The best way to get information on Usenet is not to ask a question, but to post the wrong information. -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
Le Tue, 24 Nov 2009 09:27:24 -0800, sturlamolden a écrit : > > Windows does not have daemons, so this is obviously incorrect. (There > are something called Windows Services, but multiprocessing does not use > them.) This is nitpicking. Technically it might not be a daemon but it's used as such. The important point is that it does use a (custom) form of RPC through marshalling, which is what the original question was about. -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
On 24 Nov, 16:13, Antoine Pitrou wrote: > >> Has anyone every tried wrapping the CPython lib into a daemon with an > >> RPC mechanism in order to move the GIL out of the process? > > >> I imagine this is how the multiprocessing module works. > > > It does not. > > Actually, it is how multiprocessing works under Windows (for lack of the > fork() function), except that it uses pickle by default (but it does have > xmlrpc support). Windows does not have daemons, so this is obviously incorrect. (There are something called Windows Services, but multiprocessing does not use them.) Multiprocessing on Windows uses the subprocess module. -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
Le Tue, 24 Nov 2009 02:53:30 -0800, sturlamolden a écrit : > On 22 Nov, 22:38, Patrick Stinson > wrote: > >> Has anyone every tried wrapping the CPython lib into a daemon with an >> RPC mechanism in order to move the GIL out of the process? > >> I imagine this is how the multiprocessing module works. > > It does not. Actually, it is how multiprocessing works under Windows (for lack of the fork() function), except that it uses pickle by default (but it does have xmlrpc support). Regards Antoine. -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
On Nov 24, 2009, at 5:58 , sturlamolden wrote: I have programmed parallel numerical software in Python, including on- line signal processing. I have yet to find the GIL gets in my way. Usually people complaining about the GIL does not know what they are talking about. I'd love to know which tools/libraries/approach you have found most profitable, and which were a waste of time. Seems as if many people feel the GIL gets in the way, but perhaps they've been trying methods of parallelization that just aren't as effective. What do you do? Do you have any sample code? bb -- Brian Blais bbl...@bryant.edu http://web.bryant.edu/~bblais -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
On 24 Nov, 12:15, Daniel Fetchinson wrote: > By the way, you might be interested in this thread as well: > > http://groups.google.com/group/comp.lang.python/browse_thread/thread/... It is Windows specific, and it does not work with extension modules. Yes we can embed multiple interpreters just by making multiple copies of Python26.dll. It's an ugly hack and not one I'd recommend for production servers. -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
>> icating) the multiprocessing module would be ideal. >>> > The problem is that the OP has a embedded application running threads. >>> > multiprocssing doesn't help there. >>> >>> that's right. I cannot make CPython calls from my original C-based >>> threads. >> >> >> It's quite possible to do that. A thread started from C can make >> calls to Python if it first calls PyGILState_Ensure, although you'd >> have to make sure that the Python interpreter has been previously >> initialized. See PEP 311 for details. >> >> http://www.python.org/dev/peps/pep-0311/ >> >> I also suggest that if you want people to be more receptive to write >> your replies after the quoted text. That is the custom in this >> newsgroup/mailing list. >> > > What I meant was that I am *not allowed* to make calls to the CPython > API from the threads I currently have because these threads are high > priority and are never allowed to make blocking calls. Fortunately, > each of these threads can have a completely separate interpreter, so > my idea was to create a daemon process for each thread. This daemon > would contain the libpython symbols and would make the CPython calls. > This would keep my current threads from having to contend over the > single GIL. > > My question was whether or not anyone has done anything like this in > C/C++. This situation is slightly unique in that I am trying to > maintain separate interpreters within a single app (which is currently > kind of hacked out using a single interpreter, but it's ugly), but I > could see how this sort of thing would be useful for other C/C++ apps > that implement an embedded scripting engine. > > My reference to multiprocessing was based on the idea that the library > hides the details fo the process management, shared memory, and rpc > mechanisms. Again, I can't use multiprocessing because it runs *in* > python I need this to be implemented *outside* of python to avoid > acquiring the GIL. complex, I know. > > Naturally, the most intimidating part of perusing this kind of idea is > the overhead of the processess management and the RPC. Further, > libpython executes callable objects using C function pointers, and I > can't think of a way that this would be able to re-gain access to the > original app's respective functions if the interpreter was living in > another processes. > > That's not to mention the impossibility of debugging. > > Damn you, Gil. By the way, you might be interested in this thread as well: http://groups.google.com/group/comp.lang.python/browse_thread/thread/2d537ad8df9dab67/cc4cb2b493c98170 HTH, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
On 23 Nov, 22:02, Patrick Stinson wrote: > What I meant was that I am *not allowed* to make calls to the CPython > API from the threads I currently have because these threads are high > priority and are never allowed to make blocking calls. Fortunately, > each of these threads can have a completely separate interpreter, This seems confused. What would they do with the interpreter if they cannot call the CPython API? > My question was whether or not anyone has done anything like this in > C/C++. I have programmed parallel numerical software in Python, including on- line signal processing. I have yet to find the GIL gets in my way. Usually people complaining about the GIL does not know what they are talking about. -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
On 22 Nov, 22:38, Patrick Stinson wrote: > Has anyone every tried wrapping the CPython lib into a daemon with an > RPC mechanism in order to move the GIL out of the process? > I imagine this is how the multiprocessing module works. It does not. -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
> What I meant was that I am *not allowed* to make calls to the CPython
> API from the threads I currently have because these threads are high
> priority and are never allowed to make blocking calls. Fortunately,
> each of these threads can have a completely separate interpreter, so
> my idea was to create a daemon process for each thread. This daemon
> would contain the libpython symbols and would make the CPython calls.
> This would keep my current threads from having to contend over the
> single GIL.
If this ("never allowed to make blocking calls") is a strict
requirement, then this is no solution, either. Communicating to the
remote process would involve IO, and all IO operations may block.
Of course, you might use non-blocking IO, in which case I wonder how
the thread continues if it finds that the IO is blocking. I also wonder
how the thread will find out that the result of the computation is
available, when it is not allowed to wait for the result.
In any case, I don't think you'll need a multi-process solution;
a single-process multi-threading approach will do fine. Just create
*another* thread, that runs at a low priority and is allowed to block.
Run the Python interpreter in that thread (create multiple of these if
you need them).
Then, use some queuing producer-consumer communication between the
high-priority thread and the low priority thread. Have the high
priority threads put requests into the queue, and the low priority
thread take them from the queue, dispatching them to Python. Make
sure you use non-blocking synchronization on the queue, or else
priority inheritance if you can get permission to do so.
> Damn you, Gil.
It may sound to you like the GIL is responsible for that. However,
consider a world where the GIL was removed in Python. There would
*still* be synchronization for data structures be going on. As you
are not allowed to have any blocking operation in the high-priority
threads, you *still* couldn't directly call the Python interpreter
(or any other thread-safe library) from your high-priority threads.
Regards,
Martin
--
http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
Patrick Stinson schrieb: On Mon, Nov 23, 2009 at 1:01 AM, Carl Banks wrote: On Nov 22, 10:58 pm, Patrick Stinson wrote: On Sun, Nov 22, 2009 at 3:15 PM, Diez B. Roggisch wrote: icating) the multiprocessing module would be ideal. The problem is that the OP has a embedded application running threads. multiprocssing doesn't help there. that's right. I cannot make CPython calls from my original C-based threads. It's quite possible to do that. A thread started from C can make calls to Python if it first calls PyGILState_Ensure, although you'd have to make sure that the Python interpreter has been previously initialized. See PEP 311 for details. http://www.python.org/dev/peps/pep-0311/ I also suggest that if you want people to be more receptive to write your replies after the quoted text. That is the custom in this newsgroup/mailing list. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list thanks for the tip. What I meant was that I am *not allowed* to make calls to the CPython API from the threads I currently have because these threads are high priority and are never allowed to make blocking calls. Fortunately, each of these threads can have a completely separate interpreter, so my idea was to create a daemon process for each thread. This daemon would contain the libpython symbols and would make the CPython calls. This would keep my current threads from having to contend over the single GIL. My question was whether or not anyone has done anything like this in C/C++. This situation is slightly unique in that I am trying to maintain separate interpreters within a single app (which is currently kind of hacked out using a single interpreter, but it's ugly), but I could see how this sort of thing would be useful for other C/C++ apps that implement an embedded scripting engine. AFAIK, instantiating several interpreters is possible. There seem to be problems with extension modules, but maybe you don't need them, or can work around that problem. http://mail.python.org/pipermail/python-dev/2006-December/070370.html Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
On Mon, Nov 23, 2009 at 1:01 AM, Carl Banks wrote: > On Nov 22, 10:58 pm, Patrick Stinson > wrote: >> On Sun, Nov 22, 2009 at 3:15 PM, Diez B. Roggisch >> wrote: > icating) the multiprocessing module would be ideal. >> > The problem is that the OP has a embedded application running threads. >> > multiprocssing doesn't help there. >> >> that's right. I cannot make CPython calls from my original C-based threads. > > > It's quite possible to do that. A thread started from C can make > calls to Python if it first calls PyGILState_Ensure, although you'd > have to make sure that the Python interpreter has been previously > initialized. See PEP 311 for details. > > http://www.python.org/dev/peps/pep-0311/ > > I also suggest that if you want people to be more receptive to write > your replies after the quoted text. That is the custom in this > newsgroup/mailing list. > > > Carl Banks > -- > http://mail.python.org/mailman/listinfo/python-list > thanks for the tip. What I meant was that I am *not allowed* to make calls to the CPython API from the threads I currently have because these threads are high priority and are never allowed to make blocking calls. Fortunately, each of these threads can have a completely separate interpreter, so my idea was to create a daemon process for each thread. This daemon would contain the libpython symbols and would make the CPython calls. This would keep my current threads from having to contend over the single GIL. My question was whether or not anyone has done anything like this in C/C++. This situation is slightly unique in that I am trying to maintain separate interpreters within a single app (which is currently kind of hacked out using a single interpreter, but it's ugly), but I could see how this sort of thing would be useful for other C/C++ apps that implement an embedded scripting engine. My reference to multiprocessing was based on the idea that the library hides the details fo the process management, shared memory, and rpc mechanisms. Again, I can't use multiprocessing because it runs *in* python I need this to be implemented *outside* of python to avoid acquiring the GIL. complex, I know. Naturally, the most intimidating part of perusing this kind of idea is the overhead of the processess management and the RPC. Further, libpython executes callable objects using C function pointers, and I can't think of a way that this would be able to re-gain access to the original app's respective functions if the interpreter was living in another processes. That's not to mention the impossibility of debugging. Damn you, Gil. -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
On Nov 22, 10:58 pm, Patrick Stinson wrote: > On Sun, Nov 22, 2009 at 3:15 PM, Diez B. Roggisch wrote: icating) the multiprocessing module would be ideal. > > The problem is that the OP has a embedded application running threads. > > multiprocssing doesn't help there. > > that's right. I cannot make CPython calls from my original C-based threads. It's quite possible to do that. A thread started from C can make calls to Python if it first calls PyGILState_Ensure, although you'd have to make sure that the Python interpreter has been previously initialized. See PEP 311 for details. http://www.python.org/dev/peps/pep-0311/ I also suggest that if you want people to be more receptive to write your replies after the quoted text. That is the custom in this newsgroup/mailing list. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
that's right. I cannot make CPython calls from my original C-based threads. On Sun, Nov 22, 2009 at 3:15 PM, Diez B. Roggisch wrote: > Daniel Fetchinson schrieb: >>> >>> Has anyone every tried wrapping the CPython lib into a daemon with an >>> RPC mechanism in order to move the GIL out of the process? I have >>> multiple audio threads, each of which use the python interpreter but >>> don't have to interact with each other and can might as well use a >>> separate interpreter handled by a daemon with libpython linked in. >>> >>> I've never used a C xmlrpc lib before and would probably need to set >>> up some shared memory to get it to work. Can anyone suggest a library >>> out there that would at do that sort of thing, maybe with some >>> cross-platform process management? >>> >>> I imagine this is how the multiprocessing module works. >> >> Yes, it does seem you need multiprocessing. That would take advantage >> of multiple cores and the GIL wouldn't bother you at all. Take a look >> at >> >> http://docs.python.org/library/multiprocessing.html >> >> For what you describe (separate tasks running concurrently without >> communicating) the multiprocessing module would be ideal. > > The problem is that the OP has a embedded application running threads. > multiprocssing doesn't help there. > > Diez > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
Daniel Fetchinson schrieb: Has anyone every tried wrapping the CPython lib into a daemon with an RPC mechanism in order to move the GIL out of the process? I have multiple audio threads, each of which use the python interpreter but don't have to interact with each other and can might as well use a separate interpreter handled by a daemon with libpython linked in. I've never used a C xmlrpc lib before and would probably need to set up some shared memory to get it to work. Can anyone suggest a library out there that would at do that sort of thing, maybe with some cross-platform process management? I imagine this is how the multiprocessing module works. Yes, it does seem you need multiprocessing. That would take advantage of multiple cores and the GIL wouldn't bother you at all. Take a look at http://docs.python.org/library/multiprocessing.html For what you describe (separate tasks running concurrently without communicating) the multiprocessing module would be ideal. The problem is that the OP has a embedded application running threads. multiprocssing doesn't help there. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: xmlrpc idea for getting around the GIL
> Has anyone every tried wrapping the CPython lib into a daemon with an > RPC mechanism in order to move the GIL out of the process? I have > multiple audio threads, each of which use the python interpreter but > don't have to interact with each other and can might as well use a > separate interpreter handled by a daemon with libpython linked in. > > I've never used a C xmlrpc lib before and would probably need to set > up some shared memory to get it to work. Can anyone suggest a library > out there that would at do that sort of thing, maybe with some > cross-platform process management? > > I imagine this is how the multiprocessing module works. Yes, it does seem you need multiprocessing. That would take advantage of multiple cores and the GIL wouldn't bother you at all. Take a look at http://docs.python.org/library/multiprocessing.html For what you describe (separate tasks running concurrently without communicating) the multiprocessing module would be ideal. Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown -- http://mail.python.org/mailman/listinfo/python-list
