Re: [Python-Dev] Variant of removing GIL.
On 9/17/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > Message: 9 > Date: Fri, 16 Sep 2005 21:07:23 -0500 > From: [EMAIL PROTECTED] > Subject: Re: [Python-Dev] Variant of removing GIL. > Message-ID: <[EMAIL PROTECTED]> > > > Martin> However, this is really hard to do correctly - if it were > Martin> simple, it would have been done long ago. > > I don't believe difficulty is the only (or primary) barrier. I think > *someone* would have tackled it since Greg Stein did back in 1.4(?) or his > free-threading changes would have been incorporated into the core had they > yielded speedups on multiprocessors and not hurt performance on > uniprocessors. > > Skip > It did yield speedups on multiprocessors. The uniprocessor part could've been solved just like most kernels do, one binary for UP and another for MP. That's what IBM, RedHat, Solaris, and almost all other modern kernels that support SMP machines do. In theory, if we had those changes in the CPython interpreter, we could've been running at 1.6 times the speed on dual processor machines today (according to Greg's benchmark data) and at the same speed on UP machines running the UP compiled CPython interpreter, which would not have had all the locking calls not needed on a UP machine that would hurt its performance. By now, we probably could've improved on the scalability of MP performance when running on machines with more than three processors. Mind you though, I'm not trying to oversimplify the issue. I was not using python yet at that time (I started around 1.5/1.6) and I didn't see all the info involved in the decision making process, so I'm sure there were other issues that contributed to the decision of not keeping Greg's free threading changes. My point is that not yielding speedups on multiprocessors and hurting performance on uniprocessors is not a good or valid reason to drop free-threading. -- Luis P Caamano Atlanta, GA USA ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Variant of removing GIL.
At 12:32 PM 9/17/2005 -0400, Luis P Caamano wrote: >My point is that not yielding speedups on multiprocessors >and hurting performance on uniprocessors is not a good >or valid reason to drop free-threading. It is if you have only volunteers to maintain the code base, and the changes significantly increase maintenance complexity. Also, a significant number of third-party C extensions would need to be modified for compatibility, as has already been pointed out. Note also that Jython and IronPython exist, and run on VMs that address these issues, and that the PyPy project can generate code for many kinds of backends. There's nothting stopping anybody from creating a multiprocessor-friendly backend for PyPy, for example. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Variant of removing GIL.
Luis P Caamano wrote: > Mind you though, I'm not trying to oversimplify the issue. > I was not using python yet at that time (I started around > 1.5/1.6) and I didn't see all the info involved in the decision > making process, so I'm sure there were other issues that > contributed to the decision of not keeping Greg's free > threading changes. For historical correctness, I believe there never was a decision to "not keep Greg's free threading changes". I believe Greg never actually contributed them (at least not in a publically-visible manner). This, in turn appears to be the result of the problem that nobody (including Greg) was able to tell whether the patches are actually correct (for extension modules, it appears there was agreement that the patches are *not* correct). (more correctly, it appears that some of the code made it to Python 1.5) Instead, the issue mainly died because nobody provided working code (along with a strategy on what to do with the existing extension modules). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Variant of removing GIL.
On 9/17/05, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 12:32 PM 9/17/2005 -0400, Luis P Caamano wrote: > >My point is that not yielding speedups on multiprocessors > >and hurting performance on uniprocessors is not a good > >or valid reason to drop free-threading. > > It is No, it's not because it's not true. > if you have only volunteers to maintain the code base, and the > changes significantly increase maintenance complexity. This is one very valid reason. > Also, a significant > number of third-party C extensions would need to be modified for > compatibility, as has already been pointed out. And another. > > Note also that Jython and IronPython exist, and run on VMs that address > these issues, and that the PyPy project can generate code for many kinds of > backends. There's nothting stopping anybody from creating a > multiprocessor-friendly backend for PyPy, for example. Yes, eventually when we have CPython running on more SMP or equivalent machines than UP machines, the majority will have an itch and it will get scratched. -- Luis P Caamano Atlanta, GA USA ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] os.path.diff(path1, path2)
On Fri, 16 Sep 2005, Greg Ewing wrote: > Trent Mick wrote: > > > If this *does* get added (I'm +0) then let's call it "relpath" or > > "relpathto" as in the various implementations out there: > > +1 on that, too. Preferably just "relpath". [...] +1 on adding this function, and on "relpath" as the name. I wanted this function just a few weeks ago. John ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Variant of removing GIL.
On 9/17/05, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > > Instead, the issue mainly died because nobody provided > working code (along with a strategy on what to do with > the existing extension modules). > When I first started writing python code I had just come out of about 6 years of kernel development (1994-2000). These 6 years span the times when we were making changes to UP kernels to work correctly and efficiently on SMP machines, implementing kernel threads, and thread-safing libc and other changes needed to move from the now defunct user space DCE-threads to posix threads including kernel threads and even MxN support. So, I was very familiar with thread programming. I architected, designed, and developed a bunch of distributed servers and services using python and Pyro and it all was done using nice and cool thread programming practices and techniques. It was nice to see Python supported threads. Close to a year into this when we had a lot of functionality implemented I started running scalability tests and I was shocked and appalled to find out that our architecture did not scale well on SMP machines because we depended on threads. And that's when I discovered the GIL and twisted and other process based python techniques used to get around the GIL. It was too late for us to redo things and we've been in a holding pattern since then waiting to see what will hit us first, the need to scale a lot, which means we'd have to drop some time to implement some changes to support process based scalability, or no GIL. I have my money on process based scalability. :( I'm sure that has happened to a lot of people because nobody finds about the GIL in the beginning of their python development experience. If I started writing a new complex app from scratch, now I know exactly what to do. One big problem is that a lot of people defend the GIL with the premise that the GIL is OK because we release the GIL in C extensions and during IO. That is true, the GIL is not as bad as it would've been if the CPython interpreter extensions didn't do that but it's also true that it hurts performance on SMP machines in more than one way. When I found out about this it dawn on me that the performance could've been worse although it was still way below what we expected. I don't remember exactly how but what I remember is that when I presented the problem to Guido he just told me to put up or shut up. At first I was insulted but then I realized that it was a fair response given the complexity of the problem, previous history I didn't know about, the fairly good compromise of releasing the GIL in extensions, and the lack of a large percentage of python developers asking for better performance on SMP machines. I couldn't put up so ... I think I've gone beyond my quota :-) PS The GIL is probably my only peeve about the CPython interpreter. So don't get me wrong, I love the language and I'm always grateful for all the hard work you guys put in developing such great language and implementation. -- Luis P Caamano Atlanta, GA USA ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] GIL, Python 3, and MP vs. UP (was Re: Variant of removing GIL.)
On Fri, 16 Sep 2005, "Martin v. Löwis" wrote: > Sokolov Yura wrote: > > I think I know how to remove GIL Obviously I am an idiot. > > Not an idiot, just lazy :-) Please try to implement your ideas, > and I predict that you will find: > 1. it is a lot of work to implement > 2. it requires changes to all C files, in particular to extension >modules outside the Python source tree proper. > 3. performing the conversion, even in a semi-mechanical way, will >introduce many new bugs, in the form of race conditions because >of missing locks. > > Optionally, you may also find that the performance of the > interpreter will decrease. [...] Given the points you make, and the facts that both Python 3 and real problems with continuing to scale down semiconductor chip feature sizes are on the horizon, it seems that now would be an excellent time to start work on this, with the goal of introducing it at the same time as Python 3. a. Python 3.0 will break lots of code anyway, so the extension module issue becomes far less significant. b. In x years time (x < 10?) it seems likely multiprocessor (MP) users will be in the majority. (As a result, the uniprocessor (UP) slowdown becomes less important in practice, and also Python has the opportunity of avoiding the risk of being sidelined by a real or perceived lack of MP performance.) c. Since time is needed to iron out bugs (and perhaps also to reimplememt some pieces of code "from scratch"), very early in the life of Python 3 seems like the least-worst time to begin work on such a change. I realize that not all algorithms (nor all computational problems) scale well to MP hardware. Is it feasible to usefully compile both MP and a UP binaries from one Python source code base? (I'm also quite aware that the GIL does not prevent all means of achieving efficient use of multiprocessors. I'm just concious that different parellisation problems are presumably best expressed using different tools, and that Python 3 and increased prevalance of MP systems might tip the balance in favour of removing the GIL.) Of course, it still takes a (anti-)hero to step forward and do the work... John ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] removing nested tuple function parameters
Is anyone truly attached to nested tuple function parameters; ``def fxn((a,b)): print a,b``? At one of the PyCon sprints Guido seemed okay with just having them removed when Jeremy asked about ditching them thanks to the pain they caused in the AST branch. I personally don't see them being overly useful thanks to assignment unpacking. Plus I don't think they are used very much (gut feeling, though, and not based on any grepping). Would anyone really throw a huge fit if they went away? I am willing to write a PEP for their removal in 2.6 with a deprecation in 2.5 if people are up for it. Otherwise I say they should definitely go in Python 3. -Brett ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] removing nested tuple function parameters
On Sat, Sep 17, 2005 at 06:20:08PM -0700, Brett Cannon wrote: > Is anyone truly attached to nested tuple function parameters; ``def > fxn((a,b)): print a,b``? At one of the PyCon sprints Guido seemed > okay with just having them removed when Jeremy asked about ditching > them thanks to the pain they caused in the AST branch. I personally > don't see them being overly useful thanks to assignment unpacking. > Plus I don't think they are used very much (gut feeling, though, and > not based on any grepping). > > Would anyone really throw a huge fit if they went away? I am willing > to write a PEP for their removal in 2.6 with a deprecation in 2.5 if > people are up for it. Otherwise I say they should definitely go in > Python 3. Please keep them! Twisted code uses them in places for Deferred callbacks that need to deal with multiple return values. For instance, the twisted.cred.portal.Portal's login method returns Deferred that will be called with a tuple of (interface that the avatar implements, the avatar, a 0-arg logout callable), which naturally leads to functions like: def _cbAuthenticated(self, (iface, avatar, logout)): ... Based on a quick grep, there's well over 80 uses of tuple-unpacking in function declarations in Twisted's code base. There's almost certainly many more outside of Twisted; I know I have some code at work that uses this. So the short answer is: yes, we are attached to it, we will miss it. No guarantees about throwing fits, though . -Andrew. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] removing nested tuple function parameters
Brett Cannon wrote: > Is anyone truly attached to nested tuple function parameters; ``def > fxn((a,b)): print a,b``? I find 54 instances in my Python installation. >grep -r "def.*([^=]*([^)]*,[^)]*).*):" * aifc.py:def setparams(self, (nchannels, sampwidth, framerate, nframes, comptype, compname)): bdb.py:def user_exception(self, frame, (exc_type, exc_value, exc_traceback)): binhex.py:def __init__(self, (name, finfo, dlen, rlen), ofp): cgitb.py:def html((etype, evalue, etb), context=5): cgitb.py:def text((etype, evalue, etb), context=5): compiler/ast.py:def __init__(self, (left, right), lineno=None): compiler/ast.py:def __init__(self, (left, right), lineno=None): compiler/ast.py:def __init__(self, (left, right), lineno=None): compiler/ast.py:def __init__(self, (left, right), lineno=None): compiler/ast.py:def __init__(self, (left, right), lineno=None): compiler/ast.py:def __init__(self, (left, right), lineno=None): compiler/ast.py:def __init__(self, (left, right), lineno=None): compiler/ast.py:def __init__(self, (left, right), lineno=None): compiler/ast.py:def __init__(self, (left, right), lineno=None): formatter.py:def push_font(self, (size, i, b, tt)): idlelib/Debugger.py:def show_frame(self, (frame, lineno)): imputil.py:def _process_result(self, (ispkg, code, values), fqname): inspect.py:def tokeneater(self, type, token, (srow, scol), (erow, ecol), line): lib-old/fmt.py:def text(self, (h, v), str): lib-old/fmt.py:def text(self, (h, v), text): modulefinder.py:def load_module(self, fqname, fp, pathname, (suffix, mode, type)): pdb.py:def user_exception(self, frame, (exc_type, exc_value, exc_traceback)): plat-irix5/flp.py:def create_full_form(inst, (fdata, odatalist)): plat-irix5/flp.py:def merge_full_form(inst, form, (fdata, odatalist)): plat-irix6/flp.py:def create_full_form(inst, (fdata, odatalist)): plat-irix6/flp.py:def merge_full_form(inst, form, (fdata, odatalist)): plat-mac/findertools.py:def _setlocation(object_alias, (x, y)): plat-mac/findertools.py:def _setwindowsize(folder_alias, (w, h)): plat-mac/findertools.py:def _setwindowposition(folder_alias, (x, y)): pydoc.py:def fixup((name, kind, cls, value)): pydoc.py:def __init__(self, filename, (exc, value, tb)): pydoc.py:def modpkglink(self, (name, path, ispackage, shadowed)): pydoc.py:def submodules(self, (dir, package)): pydoc.py:def isnewpackage(self, (dir, package)): sunau.py:def setparams(self, (nchannels, sampwidth, framerate, nframes, comptype, compname)): test/inspect_fodder.py:def spam(a, b, c, d=3, (e, (f,))=(4, (5,)), *g, **h): test/test_compile.py:def comp_args((a, b)): test/test_compile.py:def comp_args((a, b)=(3, 4)): test/test_compile.py:def comp_args(a, (b, c)): Binary file test/test_compile.pyc matches test/test_grammar.py:def f4(two, (compound, (argument, list))): pass test/test_grammar.py:def f5((compound, first), two): pass test/test_grammar.py:def v3(a, (b, c), *rest): return a, b, c, rest test/test_math.py:def testfrexp(name, (mant, exp), (emant, eexp)): test/test_math.py:def testmodf(name, (v1, v2), (e1, e2)): Binary file test/test_parser.pyc matches test/test_scope.py:def makeAddPair((a, b)): test/test_scope.py:def addPair((c, d)): test/test_threadsignals.py:def registerSignals((for_usr1, for_usr2, for_alrm)): threading.py:def _acquire_restore(self, (count, owner)): tokenize.py:def printtoken(type, token, (srow, scol), (erow, ecol), line): # for testing urlparse.py:def urlunparse((scheme, netloc, url, params, query, fragment)): urlparse.py:def urlunsplit((scheme, netloc, url, query, fragment)): wave.py:def setparams(self, (nchannels, sampwidth, framerate, nframes, comptype, compname)): Not to mention the dozen or so in my own codebase. STeVe -- You can wordify anything if you just verb it. --- Bucky Katt, Get Fuzzy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
