Re: Status of Python threading support (GIL removal)?
CPython itself can't... but the c extension can. Mine did. On Fri, Jun 19, 2009 at 9:50 AM, OdarR wrote: > On 19 juin, 16:16, Martin von Loewis > If you know that your (C) code is thread safe on its own, you can > > release the GIL around long-running algorithms, thus using as many > > CPUs as you have available, in a single process. > > what do you mean ? > > Cpython can't benefit from multi-core without multiple processes. > > Olivier > -- > http://mail.python.org/mailman/listinfo/python-list > -- Visit my blog at http://oddco.ca/zeroth/zblog -- http://mail.python.org/mailman/listinfo/python-list
Re: Calling subprocess with arguments
It appears to be an issue specifically with VLC, not subprocess. Thank you guys. The remote interface works through sockets, which is perfectly fine... if I create a local socket, I can have it connect to the socket with command line arguments. On Fri, Jun 19, 2009 at 9:30 AM, Tyler Laing wrote: > Sorry, XD. I'll ask the VLC people if they happen to know why VLC won't > open up the remote interface. > > -Tyler > > On Fri, Jun 19, 2009 at 9:09 AM, Mike Kazantsev wrote: > >> On Fri, 19 Jun 2009 22:00:28 +0600 >> Mike Kazantsev wrote: >> >> > On Fri, 19 Jun 2009 08:28:17 -0700 >> > Tyler Laing wrote: >> > >> > > Thanks mike, the idea that maybe some of the info isn't being passed >> is >> > > certainly interesting. >> > > >> > > Here's the output of os.environ and sys.argv: >> > > >> > ... >> > >> > I'm afraid these doesn't make much sense without the output from the >> > second results, from py itself. My suggestion was just to compare them >> > - pop the py shell, eval the outputs into two sets, do the diff and >> > you'll see it at once. >> > If there's an empty set then I guess it's pretty safe to assume that >> > python creates subprocess in the same way the shell does. >> >> Just thought of one more really simple thing I've missed: vlc might >> expect it's remote to work with tty, so when py shoves it a pipe >> instead, it automatically switches to non-interactive mode. >> >> You can remedy that a bit by superclassing subprocess.Popen, replacing >> pipes with pty, but they are quite hard to work with, prehaps pexpect >> module would be of some use there: >> >> http://pypi.python.org/pypi/pexpect/ >> >> -- >> Mike Kazantsev // fraggod.net >> >> -- >> http://mail.python.org/mailman/listinfo/python-list >> >> > > > -- > Visit my blog at http://oddco.ca/zeroth/zblog > -- Visit my blog at http://oddco.ca/zeroth/zblog -- http://mail.python.org/mailman/listinfo/python-list
Re: Calling subprocess with arguments
Sorry, XD. I'll ask the VLC people if they happen to know why VLC won't open up the remote interface. -Tyler On Fri, Jun 19, 2009 at 9:09 AM, Mike Kazantsev wrote: > On Fri, 19 Jun 2009 22:00:28 +0600 > Mike Kazantsev wrote: > > > On Fri, 19 Jun 2009 08:28:17 -0700 > > Tyler Laing wrote: > > > > > Thanks mike, the idea that maybe some of the info isn't being passed is > > > certainly interesting. > > > > > > Here's the output of os.environ and sys.argv: > > > > > ... > > > > I'm afraid these doesn't make much sense without the output from the > > second results, from py itself. My suggestion was just to compare them > > - pop the py shell, eval the outputs into two sets, do the diff and > > you'll see it at once. > > If there's an empty set then I guess it's pretty safe to assume that > > python creates subprocess in the same way the shell does. > > Just thought of one more really simple thing I've missed: vlc might > expect it's remote to work with tty, so when py shoves it a pipe > instead, it automatically switches to non-interactive mode. > > You can remedy that a bit by superclassing subprocess.Popen, replacing > pipes with pty, but they are quite hard to work with, prehaps pexpect > module would be of some use there: > > http://pypi.python.org/pypi/pexpect/ > > -- > Mike Kazantsev // fraggod.net > > -- > http://mail.python.org/mailman/listinfo/python-list > > -- Visit my blog at http://oddco.ca/zeroth/zblog -- http://mail.python.org/mailman/listinfo/python-list
Re: Calling subprocess with arguments
Thanks mike, the idea that maybe some of the info isn't being passed is
certainly interesting.
Here's the output of os.environ and sys.argv:
ty...@surak:~$ cat environ
{'XAUTHORITY': '/home/tyler/.Xauthority', 'GNOME_DESKTOP_SESSION_ID':
'this-is-deprecated', 'ORBIT_SOCKETDIR': '/tmp/orbit-tyler', 'LESSOPEN': '|
/usr/bin/lesspipe %s', 'LOGNAME': 'tyler', 'USER': 'tyler', 'PATH':
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games',
'HOME': '/home/tyler', 'WINDOWPATH': '7', 'SSH_AGENT_PID': '3235', 'LANG':
'en_CA.UTF-8', 'TERM': 'xterm', 'SHELL': '/bin/bash', 'XDG_SESSION_COOKIE':
'c40b96e13a53e01daf91b3c24a37cc3f-1245418411.623618-1455422513',
'SESSION_MANAGER': 'local/Surak:/tmp/.ICE-unix/3078', 'SHLVL': '1',
'DISPLAY': ':0.0', 'WINDOWID': '62914615', 'COLUMNS': '80',
'GPG_AGENT_INFO': '/tmp/seahorse-WkObD8/S.gpg-agent:3260:1', 'USERNAME':
'tyler', 'GDM_XSERVER_LOCATION': 'local', 'GTK_RC_FILES':
'/etc/gtk/gtkrc:/home/tyler/.gtkrc-1.2-gnome2', 'LINES': '24',
'SSH_AUTH_SOCK': '/tmp/keyring-glzV2L/socket.ssh', 'DESKTOP_SESSION':
'default', 'GDMSESSION': 'default', 'DBUS_SESSION_BUS_ADDRESS':
'unix:abstract=/tmp/dbus-HXzG5Pt75p,guid=9fc00f8a914d6f800340ca634a3b93ac',
'_': '/usr/bin/python', 'GTK_MODULES': 'canberra-gtk-module',
'GNOME_KEYRING_SOCKET': '/tmp/keyring-glzV2L/socket', 'LESSCLOSE':
'/usr/bin/lesspipe %s %s', 'GNOME_KEYRING_PID': '3065', '__GL_YIELD':
'NOTHING', 'GDM_LANG': 'en_CA.UTF-8', 'HISTCONTROL': 'ignoreboth',
'XDG_DATA_DIRS': '/usr/local/share/:/usr/share/:/usr/share/gdm/', 'PWD':
'/home/tyler', 'COLORTERM': 'gnome-terminal', 'FROM_WRAPPER': 'yes',
'LS_COLORS':
'no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.svgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:'}
ty...@surak:~$ cat argv
['./testsub.py', '-I', 'rc']
On Fri, Jun 19, 2009 at 8:21 AM, Mike Kazantsev wrote:
> On Fri, 19 Jun 2009 08:07:29 -0700
> Tyler Laing wrote:
>
> > I can't use communicate, as it waits for the child process to terminate.
> > Basically it blocks. I'm trying to have dynamic communication between the
> > python program, and vlc.
>
> Unfortunately, subprocess module doesn't allow it out-of-the-box, but
> you can use fnctl calls to perform non-blocking reads/writes on it's
> pipes, like this:
>
> flags = fcntl.fcntl(fd, fcntl.F_GETFL)
> fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
>
> After that, you can grab all the available data from the pipe at any
> given time w/o blocking.
>
> Try this recipe:
>
> http://code.activestate.com/recipes/576759/
>
> --
> Mike Kazantsev // fraggod.net
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
--
Visit my blog at http://oddco.ca/zeroth/zblog
--
http://mail.python.org/mailman/listinfo/python-list
Re: Calling subprocess with arguments
I can't use communicate, as it waits for the child process to terminate.
Basically it blocks. I'm trying to have dynamic communication between the
python program, and vlc.
On Fri, Jun 19, 2009 at 8:05 AM, Charles Yeomans wrote:
>
> On Jun 19, 2009, at 10:55 AM, Tyler Laing wrote:
>
> So no one has an answer for why passing flags and the values the flags need
> through subprocess does not work? I would like an answer. I've examined all
> the examples I could find online, which were all toy examples, and not
> helpful to my problem.
>
> On Thu, Jun 18, 2009 at 7:40 PM, Tyler Laing wrote:
>
>> I've been trying any variation I can think of to do this properly, but
>> here's my problem:
>>
>> I want to execute this command string: vlc -I rc
>>
>> This allows vlc to be controlled via a remote interface instead of the
>> normal gui interface.
>>
>> Now, say, I try this from subprocess:
>>
>> >>>p=subprocess.Popen('vlc -I rc test.avi'.split(' '), shell=False,
>> stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>
>> But I don't get the remote interface. I get the normal gui interface. So
>> how do I do it? I've tried passing ['vlc', '-I', 'rc'], I've tried ['-I',
>> 'rc'] with executable set to 'vlc'. I've had shell=True, I've had
>> shell=False. I've tried all these combinations.
>>
>> What am I doing wrong?
>>
>
> You might try
>
> p = subprocess.Popen('vlc -I rc test.avi', ...
>
> Charles Yeomans
>
>
>
--
Visit my blog at http://oddco.ca/zeroth/zblog
--
http://mail.python.org/mailman/listinfo/python-list
Re: Calling subprocess with arguments
So no one has an answer for why passing flags and the values the flags need
through subprocess does not work? I would like an answer. I've examined all
the examples I could find online, which were all toy examples, and not
helpful to my problem.
On Thu, Jun 18, 2009 at 7:40 PM, Tyler Laing wrote:
> I've been trying any variation I can think of to do this properly, but
> here's my problem:
>
> I want to execute this command string: vlc -I rc
>
> This allows vlc to be controlled via a remote interface instead of the
> normal gui interface.
>
> Now, say, I try this from subprocess:
>
> >>>p=subprocess.Popen('vlc -I rc test.avi'.split(' '), shell=False,
> stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>
> But I don't get the remote interface. I get the normal gui interface. So
> how do I do it? I've tried passing ['vlc', '-I', 'rc'], I've tried ['-I',
> 'rc'] with executable set to 'vlc'. I've had shell=True, I've had
> shell=False. I've tried all these combinations.
>
> What am I doing wrong?
>
> --
> Visit my blog at http://oddco.ca/zeroth/zblog
>
--
Visit my blog at http://oddco.ca/zeroth/zblog
--
http://mail.python.org/mailman/listinfo/python-list
Re: Status of Python threading support (GIL removal)?
This is a very long-running issue, that has been discussed many times. Here are the two sides to keeping the gil or removing it: Remove the GIL: - True multi-threaded programming - Scalable performance across a multi-core machine - Unfortunately, this causes a slow-down in single core/thread performance - Most of the slow down comes from Python's Reference counting, as every function is supposed to increase and decrease the references... which leads to a lot of synchronization and checking of locks. - It makes the core interpreter much more complex Keeping the GIL: - Processes should be used instead of threads. Look at Erlang, THE model of concurrency. It uses a process model for concurrency and message passing, rather than threads. This means you avoid deadlocks, livelocks, race conditions(not entirely, but they are reduced) - C extensions can still gain really impressive multi-threaded performance. My C extension, a wrapper around ffmpeg, releases the GIL for 90% or so of its processing time, so all other threads are still extremely responsive. - It allows python to stay simple, understandable, and offer good performance, if you know how to use it properly. Basically the two sides are that you it ends up slowing things down, it makes the interpreter more complex vs you should use processes instead, and be smart about how you use the GIL. -Tyler On Fri, Jun 19, 2009 at 2:52 AM, Jure Erznožnik wrote: > See here for introduction: > > http://groups.google.si/group/comp.lang.python/browse_thread/thread/370f8a1747f0fb91 > > Digging through my problem, I discovered Python isn't exactly thread > safe and to solve the issue, there's this Global Interpreter Lock > (GIL) in place. > Effectively, this causes the interpreter to utilize one core when > threading is not used and .95 of a core when threading is utilized. > > Is there any work in progess on core Python modules that will > permanently resolve this issue? > Is there any other way to work around the issue aside from forking new > processes or using something else? > -- > http://mail.python.org/mailman/listinfo/python-list > -- Visit my blog at http://oddco.ca/zeroth/zblog -- http://mail.python.org/mailman/listinfo/python-list
Calling subprocess with arguments
I've been trying any variation I can think of to do this properly, but
here's my problem:
I want to execute this command string: vlc -I rc
This allows vlc to be controlled via a remote interface instead of the
normal gui interface.
Now, say, I try this from subprocess:
>>>p=subprocess.Popen('vlc -I rc test.avi'.split(' '), shell=False,
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
But I don't get the remote interface. I get the normal gui interface. So how
do I do it? I've tried passing ['vlc', '-I', 'rc'], I've tried ['-I', 'rc']
with executable set to 'vlc'. I've had shell=True, I've had shell=False.
I've tried all these combinations.
What am I doing wrong?
--
Visit my blog at http://oddco.ca/zeroth/zblog
--
http://mail.python.org/mailman/listinfo/python-list
Re: GUI(eclipse+pydev/SPE) freeze when doing python auto-completion under Linux
Do you experience the same problem even on an empty program file or is it limited to just one file? -Tyler On Wed, Jun 17, 2009 at 7:47 PM, Wei, James wrote: > On Jun 18, 10:45 am, "Wei, James" wrote: > > When I am editing python program with SPE, I found that SPE will > > freeze when it is doing auto-completion. The behavior is very strange > > that I can not edit the file again. If I switch to another file and > > then switch back, I can edit it again. > > > > So I switch to eclipse+pydev, but I found the same thing happen. So I > > think it is not the problem of SPE or eclipse or pydev. > > > > If I disable auto-completion function in SPE or Eclipse+PyDev, it will > > not freeze any longer. > > > > Anybody can give me some hints? > > > > I am using Ubuntu 8.10 and updated to latest. All packages is > > installed through package manager. > > the only thing I googled related with it is > > > http://www.nabble.com/-pydev---Users--jython-2.5b1-Froze-eclipse-on-autocomplete-td21394274.html > > but I think they are different. > > -- > http://mail.python.org/mailman/listinfo/python-list > -- Visit my blog at http://oddco.ca/zeroth/zblog -- http://mail.python.org/mailman/listinfo/python-list
