Hi all,
Here is a second (last?) attempt at getting traction on fixing the GIL (is
it broken?) with BFS (WTF?).
So don't be shy (don't be too rude either) since ignoring counts as down
voting.
Relevant Python issue: http://bugs.python.org/issue7946
*Bottom line first*
I submitted an implementa
I'm interested in having *something*, but I'm not particularly
interested in the 3.x branch. I expect it to be at least 5 years before
the various Python subsystems I depend on are all ported to 3.x, and I
expect it to be at least that long before I can move onto a version of
OS X that ships with
Bill Janssen wrote:
> Right now, I'm faced with the prospect of Apple's next OS including
> Python 2.7, and being saddled with the current Python 2.x terrible
> multicore behavior for the next 5 years or so. We badly need some kind
> of patch for this in the 2.x branch.
The matter of the GIL seem
Hi,
Le dimanche 16 mai 2010 22:07:06, Nir Aides a écrit :
> *Evolving the GIL into a scheduler*
>
> The problem addressed by the GIL has always been *scheduling* threads to
> the interpreter, not just controlling access to it. The patches by Antoine
> and David essentially evolve the GIL into a s
Nick Coghlan wrote:
> Bill Janssen wrote:
> > Right now, I'm faced with the prospect of Apple's next OS including
> > Python 2.7, and being saddled with the current Python 2.x terrible
> > multicore behavior for the next 5 years or so. We badly need some kind
> > of patch for this in the 2.x bra
Bill Janssen wrote:
> As far as I'm concerned, just tying all of
> the program's threads to a single core would be fine, though I imagine
> others would differ.
Which can be done through the OS tools for setting an application's CPU
affinity.
Fixing the Python thread scheduling is only necessary
Nick Coghlan wrote:
> Bill Janssen wrote:
> > As far as I'm concerned, just tying all of
> > the program's threads to a single core would be fine, though I imagine
> > others would differ.
>
> Which can be done through the OS tools for setting an application's CPU
> affinity.
Yes, as I say, if
Bill Janssen wrote:
> Nick Coghlan wrote:
>
>> Bill Janssen wrote:
>>> As far as I'm concerned, just tying all of
>>> the program's threads to a single core would be fine, though I imagine
>>> others would differ.
>> Which can be done through the OS tools for setting an application's CPU
>> affin
Martin v. Löwis wrote:
> OSX doesn't really support thread affinity. The affinity API that they
> have is described on
>
> http://developer.apple.com/mac/library/releasenotes/Performance/RN-AffinityAPI/index.html
>
> You can't bind a thread to specific core with it, though, but you can
> reques
On Sun, May 16, 2010 at 22:52, Victor Stinner
wrote:
> I didn't followed last development around the GIL. Can you explain me why
> Python should have its own scheduler whereas each OS has already its own
> scheduler?
Because the GIL locks and unlocks threads, in practice, it already
have. But the
On Mon, 17 May 2010 08:28:08 +0200
Lennart Regebro wrote:
> But the scheduler is so simplistic it ends up fighting with the
> OS scheduler, and a large amount of CPU time is used up switching
> instead of executing.
This is already fixed with py3k.
Antoine.
On Sun, 16 May 2010 15:13:44 PDT
Bill Janssen wrote:
>
> So the patch to the threading code would presumably, for those OSs where
> the capability exists, try to put all created threads in the same
> affinity set.
This is not really a good idea. There's some code which releases the
GIL, precisel
On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote:
> On Mon, 17 May 2010 08:28:08 +0200
> Lennart Regebro wrote:
>> But the scheduler is so simplistic it ends up fighting with the
>> OS scheduler, and a large amount of CPU time is used up switching
>> instead of executing.
>
> This is already f
On Mon, 17 May 2010 14:47:25 +0200
Lennart Regebro wrote:
> On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote:
> > On Mon, 17 May 2010 08:28:08 +0200
> > Lennart Regebro wrote:
> >> But the scheduler is so simplistic it ends up fighting with the
> >> OS scheduler, and a large amount of CPU tim
Antoine Pitrou wrote:
> On Sun, 16 May 2010 15:13:44 PDT
> Bill Janssen wrote:
> >
> > So the patch to the threading code would presumably, for those OSs where
> > the capability exists, try to put all created threads in the same
> > affinity set.
>
> This is not really a good idea.
Yes, fixi
Le lundi 17 mai 2010 à 09:05 -0700, Bill Janssen a écrit :
> Antoine Pitrou wrote:
>
> > On Sun, 16 May 2010 15:13:44 PDT
> > Bill Janssen wrote:
> > >
> > > So the patch to the threading code would presumably, for those OSs where
> > > the capability exists, try to put all created threads in t
Antoine Pitrou wrote:
> Well, 2.7rc1 is scheduled in less than three weeks now. IMO any patch
> changing fundamental threading properties is a no-no (even the processor
> affinity proposal).
Unfortunately, our "fundamental threading properties" are broken for
multicore machines. And multicore s
On Mon, 17 May 2010 10:11:03 PDT
Bill Janssen wrote:
>
> I'd hate to let a fundamental flaw like this go through simply because
> someone somewhere somewhen set a completely synthetic deadline.
[...]
> I've read through that issue (several times), and I have to say that I
> wind up gnashing my te
Bill Janssen wrote:
> use it. Whether or not use explicitly know you are using threads
> (because some other package may be using them under the covers).
Of course, I meant to say, "Whether or not *youse* explicitly know you
are using threads (because some other package may be using them under
Antoine Pitrou wrote:
> Well, if instead of gnashing your teeth, you had contributed to the
> issue, perhaps a patch would have been committed by now (or perhaps
> not, but who knows?). If you stay silent, you cannot assume that
> someone else will stand up for *your* opinion (and the fact that n
On Mon, 17 May 2010 11:15:49 PDT
Bill Janssen wrote:
>
> What I did know was that
> some of our big complicated Python multi-threaded daemons had shown
> puzzling resource hogging when moved from small Macs to large 8-core
> machines with hardware RAID and lots of memory.
Could you give detailed
2010/5/16 Nir Aides
>
> *What can be done with it?*
>
> Here are some options:
> 1) Abandon it - no one is interested, yawn.
> 2) Take ideas and workarounds from its code and apply to other patches.
> 3) Include it in the interpreter as an auxiliary (turn on with a runtime
> switch) scheduler.
>
Antoine Pitrou wrote:
> On Mon, 17 May 2010 11:15:49 PDT
> Bill Janssen wrote:
> >
> > What I did know was that
> > some of our big complicated Python multi-threaded daemons had shown
> > puzzling resource hogging when moved from small Macs to large 8-core
> > machines with hardware RAID and lo
On 5/17/2010 2:59 PM, Bill Janssen wrote:
Yes, it would. As soon as I have working 3.x versions of BeautifulSoup,
PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser,
dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and
vobject, I'll let you know. :-)
There /
> Not fixing this is a big problem. It relegates the only Python which
> will be usable by many many people for many years, 2.x, to a toy
> language status on modern machines. It will have threads, but the use
> of them, either directly or indirectly by modules you may import, may
> cause unpredi
Martin v. Löwis wrote:
> > I'd hate to let a fundamental flaw like this go through simply because
> > someone somewhere somewhen set a completely synthetic deadline.
>
> No, it's not like that. We set the deadline so that we are able to
> cancel discussions like this one. It would be possible to
Martin v. Löwis wrote:
> People running into these problems will have a number of choices to
> them: switch operating systems (i.e. drop OSX for something that
> actually works), switch programming languages (i.e. drop Python for
> something that actually works), switch application architectures (i
Terry Reedy wrote:
> On 5/17/2010 2:59 PM, Bill Janssen wrote:
> > Yes, it would. As soon as I have working 3.x versions of BeautifulSoup,
> > PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser,
> > dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and
> > vobj
Antoine Pitrou wrote:
> On Sun, 16 May 2010 15:13:44 PDT
> Bill Janssen wrote:
>> So the patch to the threading code would presumably, for those OSs where
>> the capability exists, try to put all created threads in the same
>> affinity set.
>
> This is not really a good idea. There's some code wh
Florent Xicluna wrote:
> I would like to have the possibility to "./configure
> --without-broken-GIL" or "./configure --with-bfs-scheduler" in Python
> 2.7 like we "./configure --with-computed-gotos" in 3.2.
>
> It will let the opportunity for the experienced user (or the
> distribution packager)
Antoine Pitrou wrote:
> On Sun, 16 May 2010 15:13:44 PDT
> Bill Janssen wrote:
> >
> > So the patch to the threading code would presumably, for those OSs where
> > the capability exists, try to put all created threads in the same
> > affinity set.
>
> This is not really a good idea. There's so
Le lundi 17 mai 2010 à 14:36 -0700, Bill Janssen a écrit :
>
> Could the macro that releases the GIL also release the thread affinity?
We release the GIL in a variety of situations which don't necessarily
involve heavy computations (such as: waiting for IO or sleeping).
Furthermore, having severa
> But some of these, like JCC+pylucene, nltk, and vobject, were
> developed with idiosyncratic funding resources which no longer exist.
> Others, like pyglet, were developed for a completely different
> purpose, and I doubt the developers care what I want. So,
> realistically, I doubt it will be l
I would like to restart this thread with 2 notes from the lively discussion:
a) Issue 7946 (and this discussion?) concerns Python 3.2
b) The GIL problems are not specific to OSX. The old and new GIL misbehave
on GNU/Linux and Windows too.
[Putting on anti-frying-pan helmet]
Nir
_
On Mon, May 17, 2010 at 15:05, Antoine Pitrou wrote:
> On Mon, 17 May 2010 14:47:25 +0200
> Lennart Regebro wrote:
>> On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote:
>> > On Mon, 17 May 2010 08:28:08 +0200
>> > Lennart Regebro wrote:
>> >> But the scheduler is so simplistic it ends up figh
On Tue, 18 May 2010 08:45:41 +0200
Lennart Regebro wrote:
> >>
> >> Are you referring to the "New GIL"?
> >
> > Yes.
>
> At has been shown, it also in certain cases will race with the OS
> scheduler, so this is not already fixed, although apparently improved,
> if I understand correctly.
"Race"
Nir Aides wrote:
> I would like to restart this thread with 2 notes from the lively discussion:
>
> a) Issue 7946 (and this discussion?) concerns Python 3.2
> b) The GIL problems are not specific to OSX. The old and new GIL
> misbehave on GNU/Linux and Windows too.
I think Antoine and Bill went o
Bill Janssen, 17.05.2010 23:09:
Most folks don't use "threads"
Seems like a somewhat reasonable assumption to me.
they use a higher-level abstraction like the nltk library.
I have my doubts that this applies to "most folks" - likely not even to
most of those who use threads.
Stefan
___
On Tue, May 18, 2010 at 12:53, Antoine Pitrou wrote:
> "Race" is a strange term here and I'm not sure what you mean.
> The issue found out by Dave Beazley can't be reasonably described by
> this word, I think.
OK, maybe "race" is the wrong word. But that doesn't mean the issue
doesn't exist.
> P
Le mardi 18 mai 2010 à 14:16 +0200, Lennart Regebro a écrit :
> > Please read and understand the issue report mentioned by Nir before
> > trying to make statements based on rumours heard here and there.
>
> Oh, so Dave Beazleys reports is a rumour now.
Your and other people's grandiloquent inter
On Tue, May 18, 2010 at 14:52, Antoine Pitrou wrote:
>
> Le mardi 18 mai 2010 à 14:16 +0200, Lennart Regebro a écrit :
>> > Please read and understand the issue report mentioned by Nir before
>> > trying to make statements based on rumours heard here and there.
>>
>> Oh, so Dave Beazleys reports i
Lennart Regebro wrote:
> On Tue, May 18, 2010 at 14:52, Antoine Pitrou wrote:
>> Le mardi 18 mai 2010 à 14:16 +0200, Lennart Regebro a écrit :
Please read and understand the issue report mentioned by Nir before
trying to make statements based on rumours heard here and there.
>>> Oh, so D
On Tue, May 18, 2010 at 3:43 PM, "Martin v. Löwis" wrote:
> So please join us in considering the issue fixed unless you can provide
> a really world example that demonstrates the contrary.
The server software I maintain (openrpg) experiences this issue with
when I tried porting the server code to
On Sun, May 16, 2010 at 1:07 PM, Nir Aides wrote:
> Relevant Python issue: http://bugs.python.org/issue7946
Is there any chance Antoine's "gilinter" patch from that issue might
be applied to python 2.7? I have been experiencing rare long delays
in simple io operations in multithreaded python ap
On Tue, 18 May 2010 14:39:43 -0700
Mike Klaas wrote:
> On Sun, May 16, 2010 at 1:07 PM, Nir Aides wrote:
>
> > Relevant Python issue: http://bugs.python.org/issue7946
>
> Is there any chance Antoine's "gilinter" patch from that issue might
> be applied to python 2.7? I have been experiencing r
On Tue, May 18, 2010 at 2:50 PM, Antoine Pitrou wrote:
> There's no chance for this since the patch relies on the new GIL.
> (that's unless there's a rush to backport the new GIL in 2.7, of course)
Thanks I missed that detail.
> I think your "rare long delays" might be related to the old GIL's
On Tue, 18 May 2010 17:26:44 -0700
Mike Klaas wrote:
>
> > I think your "rare long delays" might be related to the old GIL's own
> > problems, though. How long are they?
>
> Typically between 20 and 60s.
You mean milliseconds I suppose?
If it's the case, then you may simply be witnessing garbag
> I think the new GIL should be given a year or so in the wild before
> you start trying to optimize theoretical issues you may run into. If
> in a year people come back and have some examples of where a proper
> scheduler would help improve speed on multi-core systems even more,
> then we can addr
> From: "Martin v. L?wis"
> To: Dj Gilcrease
> Cc: python-dev@python.org
> Subject: Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
> Message-ID: <4bf385e3.9030...@v.loewis.de>
> Content-Type: text/plain; charset=ISO-8859-1
>
>> I think the
Does anybody think that by having problems with the new GIL that it might
further weaken the adoption rate for 3k? -peter
On 5/19/10 7:00 AM, "David Beazley" wrote:
>> From: "Martin v. L?wis"
>> To: Dj Gilcrease
>> Cc: python-dev@python.org
>> Subj
Peter Portante wrote:
> Does anybody think that by having problems with the new GIL that it might
> further weaken the adoption rate for 3k? -peter
No, to the contrary. By having the new GIL being superior to the old
implementation, the adoption rate for 3k will increase.
Regards,
Martin
On Wed, May 19, 2010 at 4:17 PM, Peter Portante
wrote:
> Does anybody think that by having problems with the new GIL that it might
> further weaken the adoption rate for 3k? -peter
Nope, because the remaining issues with the new GIL affect the old GIL
as well, and have yet to be proven to affect
Yes, but the million dollar question is whether or not it really is superior
with the I/O convoying problem in the current implementation (an effect that is
substantially worse with the new GIL than with the old one by the way).
Personally, I think the convoying issue is something that will hav
Martin v. Löwis wrote:
> Peter Portante wrote:
> > Does anybody think that by having problems with the new GIL that it might
> > further weaken the adoption rate for 3k? -peter
>
> No, to the contrary. By having the new GIL being superior to the old
> implementation, the adoption rate for 3k wil
54 matches
Mail list logo
