Re: Looping [was Re: Python and the need for speed]
On Wed, 11 Oct 2017 10:57 pm, Stefan Ram wrote:
> FWIW, in is book "Touch of Class" (2009) Bertrand Meyer writes:
>
> |Such instructions are just the old goto in sheep's clothing.
> |Treat them the same way as the original:
> |
> |/Touch of Methodology/:
> | Sticking to one-entry, one-exit building blocks
> |Stay away from any "break" or similar control mechanism.
I have a great deal of respect for Meyer, but in this case I think he has it
badly wrong on both counts (`break` and "one-entry, one-exit").
Unrestricted GOTO (as in BASIC, circa 1970, where you could jump to any line
in the code, or as in assembly) is rightly considered harmful (albeit
necessary in assembly). But if Meyer means *any* sort of jump ("similar
control mechanism"), then that would rule out not just `break` and `continue`
but also:
* while loops
* for loops
* if...else
* case/switch statements
* calling subroutines (functions or procedures)
* exception handling
Remember that a function call is basically a GOTO in disguise: execution jumps
to the function, and jumps back when the function returns. In BASIC, there
was a GOSUB intermediate between a GOTO and a procedure call.
Surely Meyer doesn't mean to say we should never call functions or use
`while`, `for` or `if`. So he has to distinguish between kinds of jumps:
Good jumps (I presume):
* function calls
* if...else
* looping
Evil jumps:
* unrestricted BASIC-style GOTO/GOSUB any line number
* break/continue
Not sure:
* GOTO where there are restrictions on where you can jump
* COMEFROM
(I kid: I'm sure Meyer would oppose COMEFROM, and I expect that even
Pascal-style restricted GOTO would be on his "evil" list to avoid.)
So the question becomes, why are such harmless, simple to understand,
innocuous jumps like `break` and `continue` in the evil list, when they not
only simplify code but make it more efficient?
# with break
for i in range(2**64):
if isprime(i):
print(i, "is prime")
break
# without break
still_searching = True
for i in range(2**64):
if still_searching and isprime(i):
print(i, "is prime")
still_searching = False
# without break, attempt number 2
still_searching = True
i = 0
while still_searching and i < 2**64:
if isprime(i):
print(i, "is prime")
still_searching = False
Unrestricted jumps *into* a subroutine are hard to reason about. In general,
subroutines should have a single entry point. But the requirement for one
exit is too strict. From the caller's perspective, there is no way to tell
how many exit point a function has: subroutines are black boxes that the
caller cannot see into, and the difference between a single exit and multiple
exits is invisible.
But from the point of view of the subroutines, the rule "one exit" is like the
rule "no break" for loops: it makes the code more complex and less efficient.
If you're done, you're done, and you might as well return out of the function
rather than write boilerplate code to pad it out until you get to the very
end. With only a single condition to test, there's not much difference
between the two:
# with single exit # with multiple exits
def foo(): def foo():
if condition: if condition:
result = 1 return 1
else: return 2
result = 2
return result
but as the number of decision points increase, the complexity required to keep
a single exit also increases.
Ironically, after telling us to stick to code with one entry and one exit,
Meyer then contradicts himself by recommending exceptions:
> |You can use exception handling as a technique of last resort
> |to handle unexpected events for which the normal control
> |structures let you down.
Even more ironically, exception handling is most similar to a COMEFROM, which
was invented as a joke.
--
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
Gregory Ewing writes: > A JIT compiler works by observing the actual values To be pedantic, that's called a "tracing JIT". Other runtime code generation is also frequently called JIT compilation even when it's fairly stupid combining of assembly code templates, or the like. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Chris Angelico writes: > while True: > c = sys.stdin.read(1) > if not c: break > if c.isprintable(): text += c > elif c == "\x08": text = text[:-1] > # etc > Can you write _that_ as a do-while? I prefer to write that sort of thing with iterators: for c in iter(lambda: sys.stdin.read(1), ''): if c.isprintable(): text.append(c) elif c == '\x08': text.pop() ... -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 19/04/2017 17:23, Marko Rauhamaa wrote: bartc : Enough works in that 'pcc' project, in terms of file i/o, that it can still run plenty of useful programs, such as compilers. This might have come up before, but do you have a language specification somewhere? (Nothing formal nor up-to-date. (In the absence of other users, or enough people using programs written in it in used by others, then it is easy for the project to be volatile. Previous, simpler versions I used in the 90s, which were part of a graphics app, /were/ used by others, both the language and the programs I and they wrote. So it had to be more stable. It even had a 350-page manual, created with the same graphics app of course.) I'll try and throw something together, because it could do with it.) -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
bartc : > Enough works in that 'pcc' project, in terms of file i/o, that it can > still run plenty of useful programs, such as compilers. This might have come up before, but do you have a language specification somewhere? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 19/04/2017 15:35, Chris Angelico wrote: On Wed, Apr 19, 2017 at 11:46 PM, bartc wrote: You'd be surprised how easy it is to be non-OS-neutral. I misread that as 'easy to be OS-neutral'. If I turn it around, you're saying it is easy to be OS-specific. But we know that! And that is the problem. It's not so simple. By OS-neutral I mean code that doesn't depend on special features of either OS (Ie. Windows and Linux). Not conditional code that does either Windows stuff or Linux stuff. Which means, therefore, that you either can't have any file system facilities, or must transparently expose them to the application. You also will have a lot of trouble making cross-platform subprocess functionality. Even sockets, which are notionally the same on all platforms, have a lot of little differences. To be truly OS-neutral, you basically have to be either nerfed to nothing, or so thin you're doing nothing. Enough works in that 'pcc' project, in terms of file i/o, that it can still run plenty of useful programs, such as compilers. (See from line 3357 of pcc64.c for example; a list of functions that are patched in to the host rather linked to an external msvcrt or libc.so shared library.) In this project, the only OS-specifics it really needs are: Windows: use LoadLibrary/GetProcAddr, and msvcrt.dll Linux: use dlopen/dlsym, and libc.so.6 which wouldn't be hard to arrange. Any other specifics are in the domain of the user programs (different libraries for screen i/o for example). However, getting it to choose between one of those two pairs of functions is trivial, compared with the dependencies of a a typical open source project. (I've just downloaded CPython sources from your github link, and the first thing I see is an 18,000 line 'configure' script (somewhat bigger than my entire C compiler). This doesn't run on Windows, not without importing most of Linux via cygwin or MSYS. But the sources do now appear to support a Windows build: Step 1: Download Visual Studio Express 2015 Last time I looked, it seemed to be a 5GB to 11GB download (GB not MB!). Nevertheless I start the process to get a better idea. It says: "Visual Studio requires .NET Framework 4.6 or higher." Here where I start to think it might be easier to download all the Linux stuff after all! As, with my track record, I could download tons of stuff, and it still won't work. FWIW, my mcc64.c file compiles to a 0.2MB executable which is capable of compiling that pcc64.c interpreter (I think it does it in 50msec for good measure; Tiny C will also do it instantly). /That/ is how simple things should be, and can be. Cue a million excuses for why things have to be the way they are...) -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Wed, Apr 19, 2017 at 11:46 PM, bartc wrote: >> You'd be surprised how easy it is to be non-OS-neutral. > > It's not so simple. By OS-neutral I mean code that doesn't depend on special > features of either OS (Ie. Windows and Linux). Not conditional code that > does either Windows stuff or Linux stuff. > Which means, therefore, that you either can't have any file system facilities, or must transparently expose them to the application. You also will have a lot of trouble making cross-platform subprocess functionality. Even sockets, which are notionally the same on all platforms, have a lot of little differences. To be truly OS-neutral, you basically have to be either nerfed to nothing, or so thin you're doing nothing. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 19/04/2017 12:27, Chris Angelico wrote: On Wed, Apr 19, 2017 at 8:33 PM, bartc wrote: [Warning: this is nothing to do with Python.] My interpreter is on github as /one/ C source file (a link would be inappropriate here). People can compile it with -O3 or -O2 if they wish. It's a bit simpler than building CPython, and OS-neutral; that was deliberate. Then send me a link, and I'll try to compile it. You'd be surprised how easy it is to be non-OS-neutral. Have you compiled it on the three major platforms of today (Lin/Win/Mac)? It should work on Windows and Linux, but I've only tested with gcc and Tiny C recently. I don't have a Mac. The original link was for 32-bits, as that tended to be faster, but I've redone versions for both: click on either pcc32.c or pcc64.c here (and do what you have to do to download to your machine): https://github.com/bartg/langs/tree/master/qlang Build instructions are in the sources, but are basically just: gcc pcc64.c -opcc -lm Test using: ./pcc hello # .q is assumed I provide hello.q, or you can key it in; create a file hello.q with: proc start = println "Hello, World" end To test on something bigger, download mc.qa (a composite of 27 .q files), and try that: ./pcc mc.qa (mc.qa is an entire compiler for my static language; it should (byte-code) compile instantly then give usage instructions. To try it, copy hello.q to hello.m - the two languages conveniently have the same syntax - then try: ./pcc mc.qa hello # defaults to .asm output I think ./pcc mc.qa /c64 hello # hello.m to hello.c I think it took 10ms to compile the 23Kloc of mc.qa to byte-code on my current machine.) If you want another example of a trivial-to-build project, try: https://github.com/bartg/langs/blob/master/bccproj/mcc64.c This is my C compiler in one source file (although not finished). Also OS-neutral (to run; however generated code runs on Win64). This is how I wish other people would distributed their open source projects! > You'd be surprised how easy it is to be non-OS-neutral. It's not so simple. By OS-neutral I mean code that doesn't depend on special features of either OS (Ie. Windows and Linux). Not conditional code that does either Windows stuff or Linux stuff. This means the 'pcc' program above is restricted (it needs either LoadLibrary/GetProcAddr or dlopen/dlsym to be able to access external libraries, although essential functions are patched in). -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Op 16-04-17 om 19:07 schreef Terry Reedy: > On 4/16/2017 11:35 AM, Michael Torrie wrote: >> On 04/16/2017 07:57 AM, bartc wrote: >>> But people just don't want it. >>> >>> /That/ is what surprises me, when people reject things that to me are >>> no-brainers. > > Whereas to me, it is a no-brainer that we are better off *without* > multiple while/loop constructs. > >> I simply don't care about these missing loop constructs. > > I do ;-) I consider the current simplicity a feature. > > > Python works >> great for what I use it for, and apparently works well for many people. > > The great majority* of 'repetition with variation' is sequentially > processing items from a collection. Python does that nicely with 'for > item in collection: process(item)'. While-loops take care of > everthing else. Not really, unless you count on the break statement. But if you count on that, you don't even need a while, you can start a for loop with a generator that never stops and use breaks. There was a time something like the following was seriously considered for introduction in the language. do part1 while condition: part2 which would be equivalent to the following: while True: part1 if not condition: break part2 But suddenly this was no longer considered. I still wish they had followed through. I think such a construct comes up often enough, to have such a loop construct. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
Chris Angelico : > On Wed, Apr 19, 2017 at 8:33 PM, bartc wrote: > You'd be surprised how easy it is to be non-OS-neutral. Have you > compiled it on the three major platforms of today (Lin/Win/Mac)? Generally, I don't try to be OS-neutral because 1. I need it for Linux only 2. I don't have Windows and Mac machines to test stuff on 3. Windows and Linux are completely different system-programming-wise Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Wed, Apr 19, 2017 at 8:33 PM, bartc wrote: > My interpreter is on github as /one/ C source file (a link would be > inappropriate here). People can compile it with -O3 or -O2 if they wish. > It's a bit simpler than building CPython, and OS-neutral; that was > deliberate. Then send me a link, and I'll try to compile it. You'd be surprised how easy it is to be non-OS-neutral. Have you compiled it on the three major platforms of today (Lin/Win/Mac)? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 19/04/2017 01:07, Erik wrote: On 19/04/17 00:33, bartc wrote: [Talking about an interpreter that is /not/ for Python] With the sort of lower level programs I write (in another dynamic language not Python), such an assembly layer improved performance 2-3 times over using 100% HLL compiled using C and gcc-O3. Did you give the C compiler enough hints though? If writing native C then I can spend a lot of time tweaking different things, and probably I could get a worthwhile improvement. But I doubt it would be double the speed. But I anyway generate the C code from another language and I don't want it tied to a particular compiler or be full of specific pragmas and such, unless necessary. I don't need to because I can always bring out the asm version! [However that only works on Win64 platform. I use C for portability.] Also, remember that -O3 might (and by that I mean probably will! ;)) make your code larger. If you have some specific core areas of your interpreter that are now large enough to cause instruction cache misses then a smaller -O2 (or even -Os) compiled version might perform better on your hardware. My interpreter is on github as /one/ C source file (a link would be inappropriate here). People can compile it with -O3 or -O2 if they wish. It's a bit simpler than building CPython, and OS-neutral; that was deliberate. -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 19/04/17 00:33, bartc wrote: So that's 'label-pointers' which I assume must correspond to computed goto. Yes - just different terminology. Being able to take the address of a label and "goto address" rather than "goto label". (I don't know why they should be faster than a switch; they just are.) In C, the code generated for a switch() can do almost anything. It might cache the expression in a temporary and generate a linear "if .. else if .. else if ... else if" sequence (which is probably quite common for a sparsely populated set of values, after an initial range check is done) or it might generate a jump table (similar to the computed gotos) (which is probably quite common for a densely populated set of values, after an initial range check is done and an adjustment to make the index zero-based). It could also generate code that is some sort of hybrid of the two (for example, a switch with several densely-populated areas in an otherwise sparse set of values might do linear range-checks to find the right jump-table to use for each dense area). What the computed goto stuff does is effectively reduce all of that to a single jump table with a known set of indices - there are 256 opcodes starting from 0 and each has an entry in an array of code pointers. To jump to the handler for an opcode, just 'goto handlers[op]'. No range checking, no index adjustment, no linear test sequences. This is what makes the dispatch fast. With the sort of lower level programs I write (in another dynamic language not Python), such an assembly layer improved performance 2-3 times over using 100% HLL compiled using C and gcc-O3. Did you give the C compiler enough hints though? Much like the above computed-goto stuff (which is less of a hint and more of an instruction that the compiler should be using jump tables and nothing else for that bit of code) there are lots of other ways of spelling things differently in C that can give better performance than what you might get by default (and I'm not even talking about compiler-specific #pragmas or whatever). Also, remember that -O3 might (and by that I mean probably will! ;)) make your code larger. If you have some specific core areas of your interpreter that are now large enough to cause instruction cache misses then a smaller -O2 (or even -Os) compiled version might perform better on your hardware. E. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 19/04/2017 00:19, Gregory Ewing wrote: bartc wrote: Another optimisation for something like ADD, was to to check for very common types such as INT, and deal with those locally. And then you're on the path to writing a JIT compiler. If you can identify such cases, you might as well generate machine code for them. No, the fast code path to be executed already exists when writing the interpreter. The decision to use it would be done on a per-byte-code basis. Rather a crude way of speeding things up. If you're ADDing something that isn't two ints or is just a bit too complex (int+float for example), then that would slow down because of the extra check for two ints. Where it gets more JITish is if it identifies that a sequence of byte-codes can be handled by dedicated code synthesised at runtime, which can also have hard-coded operands rather than picking them up from the byte-code stream. But this can still be done during an initial pass. This is some way from the very ambitious tracing JITs that people are up to now. (I'm not going down route because I specialise in doing things the easy way.) -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
Steve D'Aprano wrote: You seem to be describing a *tracing JIT* compiler. Well, yes, but it seems to me that a JIT compiler that *doesn't* use tracing is just an AOT compiler that you happen to run immediately before executing the program. Cython doesn't do any of that -- it's just a plain, boring, standard ahead-of-time compiler that goes by the type info you give it and nothing more. That cannot be the full story, because Cython can optimize some regular, unannotated Python code. Maybe, but the point is that it does it using only information present in the source. Also, it's usually run once ahead of time to generate object code that's stored. I think we may be using different definitions of "JIT". If you define it simply as running the compiler just before you execute the code, then *some* ways of using Cython might *barely* qualify -- you can set things up so that a Python import will trigger compiling Cython code if necessary. But that doesn't seem like a useful way of defining JIT to me. Such a compiler is not a different kind of beast from an AOT compiler. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 18/04/2017 23:33, Erik wrote: On 18/04/17 11:30, bartc wrote: On 18/04/2017 10:32, Erik wrote: the improvements over the original huge switch() to dispatch the bytecodes to the correct handler appear to have made this type of optimization less effective. What did they do to it, and on which version? It's the computed 'goto' stuff that I'm referring to. At the time, I was looking at 2.7.9 (which just has the big switch) and 3.5.0 (which has the computed gotos). So that's 'label-pointers' which I assume must correspond to computed goto. (I don't know why they should be faster than a switch; they just are.) Perhaps the Windows build process requires that things are stable and doesn't support this type of development at all. You should take this part of the discussion to python-dev if you want to be able to build and experiment with it on Windows. (OK, I've complained enough times in other groups about open source projects being too Linux-centric.) If that had worked, then further optimisations are possible, such as doing a pre-pass combining common operations, that would not be worthwhile using 'official' byte-codes.) That is effectively what my experiments were doing - sort of -, but without introducing an assembler layer. I'm not convinced about that - given the right hints, a half-decent C compiler these days will produce pretty good code. With the sort of lower level programs I write (in another dynamic language not Python), such an assembly layer improved performance 2-3 times over using 100% HLL compiled using C and gcc-O3. In fact the rest of the code didn't need such a hot compiler and didn't need to be in C. With less numeric but still internally intensive tasks such as compilers, using the assembly layer was still over 50% faster than using C and gcc-O3. However, with the very high level programs that many people (not me!) like to write in Python, any such advantages would narrow I think. -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 19/04/17 00:08, Gregory Ewing wrote: Erik wrote: When considering special-casing this opcode sequence, remember that in-place operations can be performed on anonymous objects (i.e., those referenced by a collection and not bound directly to a namespace): I think this means you would want multiple versions of the +=1 opcode: ADD_ONE_TO_NAME, ADD_ONE_TO_ITEM, ADD_ONE_TO_ATTR, etc. Essentially one for each variant of the STORE operation. Yes, I get that ;) That's what I meant by a "set of opcodes" - some of which know that there's an implicit ROT3 to do. Perhaps in some other circumstances there's a ROT3 and another op before the STORE. Or perhaps three other ops ... I'm just pointing out that concentrating on folding a very specific sequence of opcodes that you happen to see in your test code disassembly does not necessarily scale well. Why just INPLACE_ADD? For each variant of each type of STORE operation (we know there are at least two), you also might want to have MULTIPLY, SUBTRACT, DIVIDE, BIT_OR, BIT_AND, SHIFT_LEFT, SHIFT_RIGHT, EXP and MODULO versions ... (have I forgotten any?) E. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 19/04/2017 00:08, Gregory Ewing wrote: Erik wrote: When considering special-casing this opcode sequence, remember that in-place operations can be performed on anonymous objects (i.e., those referenced by a collection and not bound directly to a namespace): I think this means you would want multiple versions of the +=1 opcode: ADD_ONE_TO_NAME, ADD_ONE_TO_ITEM, ADD_ONE_TO_ATTR, etc. Essentially one for each variant of the STORE operation. This where it starts to get out of hand, if using a fixed set of byte-codes. I suggested earlier in the thread using special references, which can be created for each kind of access (GLOBAL, FAST, INDEX etc). But then you have one REF type and only need ADD_ONE_TO_REF. Or you might choose to optimise just the local name case, on the basis that the overhead of things like item and attribute lookup would probably swamp any possible gains. But you would reduce two lookups to one, so the gains might be greater with those more elaborate accesses. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
bartc wrote: But another problem was, there /are/ no simple byte-codes in CPython! Yes, that's the crux of the whole thing. It's the reason it's so hard to compile Python to anything that runs significantly faster than interpreted code. Another optimisation for something like ADD, was to to check for very common types such as INT, and deal with those locally. And then you're on the path to writing a JIT compiler. If you can identify such cases, you might as well generate machine code for them. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
Erik wrote: When considering special-casing this opcode sequence, remember that in-place operations can be performed on anonymous objects (i.e., those referenced by a collection and not bound directly to a namespace): I think this means you would want multiple versions of the +=1 opcode: ADD_ONE_TO_NAME, ADD_ONE_TO_ITEM, ADD_ONE_TO_ATTR, etc. Essentially one for each variant of the STORE operation. Or you might choose to optimise just the local name case, on the basis that the overhead of things like item and attribute lookup would probably swamp any possible gains. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Ben Bacarisse wrote: I fond the proportion on while True: loops surprising. Is there something about Python that encourages that kind of loop? Maybe because for-loops take care of most of the ordinary cases in Python, leaving while-loops to cover the weird ones, many of which need one or more exits in the middle somewhere. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 18/04/17 11:30, bartc wrote: On 18/04/2017 10:32, Erik wrote: the improvements over the original huge switch() to dispatch the bytecodes to the correct handler appear to have made this type of optimization less effective. What did they do to it, and on which version? It's the computed 'goto' stuff that I'm referring to. At the time, I was looking at 2.7.9 (which just has the big switch) and 3.5.0 (which has the computed gotos). I can't be any more specific than that without looking at commit histories - you can do that if you want ;). I am running on Linux using GCC, so I am using the computed-goto version (I hadn't realised at the time that it was optional - so perhaps my changes _would_ still have shown an improvement on builds that still use the big switch ... I'll have to look again when I have time). The opcodes have changed from bytecode to wordcode since then, so my changes would need looking at again anyway. (I couldn't get anywhere with CPython because: [snip] It obviously builds on Windows (isn't there a MSVC project?), but FWIW, on Linux things do get a bit hairy when you want to introduce new opcodes, for example (IIRC, the build process itself uses a version of Python that it builds to generate some "frozen" .py files for the final build and that can cause things to get out of step where the build-process executable doesn't understand the new opcodes - I forget the details, but it's something like that. I think the answer was to keep "clean"ing the build area and building from scratch each time the opcode map changes). Perhaps the Windows build process requires that things are stable and doesn't support this type of development at all. You should take this part of the discussion to python-dev if you want to be able to build and experiment with it on Windows. If that had worked, then further optimisations are possible, such as doing a pre-pass combining common operations, that would not be worthwhile using 'official' byte-codes.) That is effectively what my experiments were doing - sort of -, but without introducing an assembler layer. I'm not convinced about that - given the right hints, a half-decent C compiler these days will produce pretty good code. E. -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Tue, 18 Apr 2017 03:43 pm, Gregory Ewing wrote: > Steve D'Aprano wrote: >> I'm not sure why the Cython devs maintain this is not a JIT compiler. >> Perhaps I misunderstand something. > > A JIT compiler works by observing the actual values taken on > by variables at run time, and if it notices that a particular > variable seems to always have a particular type, it compiles > a special version of the code to handle that type, protected > by appropriate guards. You seem to be describing a *tracing JIT* compiler. > Cython doesn't do any of that -- it's just a plain, boring, > standard ahead-of-time compiler that goes by the type info > you give it and nothing more. That cannot be the full story, because Cython can optimize some regular, unannotated Python code. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 18/04/2017 10:32, Erik wrote: FWIW, I spent some time about a year ago looking at things like this (small improvements to the peephole optimizer which allowed certain very common sequences to be folded into a (new) opcode which in turn allowed other optimizations to avoid branching). The changes worked, but didn't actually improve performance significantly in my tests (which is why I ended up not bothering to propose anything). I remember back in the day (circa 1.5.2?) that trips-around-the-interpreter-loop were significant and avoiding them could give wins. However, in the current CPython interpreter, the improvements over the original huge switch() to dispatch the bytecodes to the correct handler appear to have made this type of optimization less effective. What did they do to it, and on which version? I've just downloaded 3.6.1 and it was about 17% faster than 3.4.3 on running that test I posted a few days ago. This was on Windows. But I also looked at CPython a couple of years back with a view to doing something with it. I didn't manage anything, but learned that the gcc-compiled Linux version enabled label-pointers in place of the big switch, while the MSVC-compiled Windows version still used the switch. The differences (on small integer benchmarks) were also 15% or so. (I couldn't get anywhere with CPython because: (1) It's so complicated that I could only build on Linux (2) It depends on configure scripts and makefiles that I have no idea about (3) None of my tools (editors etc) work on Linux so it was awkward (although cross-OS working using a real Windows and a virtual Linux would have been viable I suppose.) (4) But the project was just too complicated to find my way around. It would have taken ages (the source is full of conditional blocks of code, and uses macros everywhere). The thing I had in mind, that I've used elsewhere, was to add an extra layer to the byte-code dispatcher, written in assembly in its own module. This has a dispatch loop (using 'threaded' code and with register-resident interpreter variables) which intercepts each byte-code, looks at it, and passes it on to the standard dispatcher if it can't deal with it. So initially it would slow it down. But then you gradually add local handling of simple byte-codes, and eventually it gets faster than running the 100% HLL version. But another problem was, there /are/ no simple byte-codes in CPython! Or very few (jump-absolute for example). Another optimisation for something like ADD, was to to check for very common types such as INT, and deal with those locally. But the handling required in the C code even for that looked horrendous. You need to be able to able to handle such cases in a dozen lines of assembly, otherwise the code gets impractical. If that had worked, then further optimisations are possible, such as doing a pre-pass combining common operations, that would not be worthwhile using 'official' byte-codes.) -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 13/04/17 18:50, MRAB wrote:
On 2017-04-13 09:08, Steven D'Aprano wrote:
On Wed, 12 Apr 2017 16:30:38 -0700, bart4858 wrote:
Is it possible to skip the STORE_NAME op-code? If you knew *for sure*
that the target (x) was a mutable object which implemented += using an
in-
place mutation, then you could, but the only built-in where that applies
is list so even if you could guarantee x was a list, it hardly seems
worth the bother.
If the reference to be stored by STORE_NAME is the same as the reference
returned by LOAD_NAME, then STORE_NAME could be omitted.
That would just mean remembering that address.
When considering special-casing this opcode sequence, remember that
in-place operations can be performed on anonymous objects (i.e., those
referenced by a collection and not bound directly to a namespace):
>>> import dis
>>> dis.dis(compile("x = [0, 1, 2]; x[1] += 1;", "", "single"))
1 0 LOAD_CONST 0 (0)
3 LOAD_CONST 1 (1)
6 LOAD_CONST 2 (2)
9 BUILD_LIST 3
12 STORE_NAME 0 (x)
15 LOAD_NAME0 (x)
18 LOAD_CONST 1 (1)
21 DUP_TOP_TWO
22 BINARY_SUBSCR
23 LOAD_CONST 1 (1)
26 INPLACE_ADD
27 ROT_THREE
28 STORE_SUBSCR
29 LOAD_CONST 3 (None)
32 RETURN_VALUE
So in this case, the STORE_SUBSCR does the re-binding, but it is
separated from the INPLACE_ADD by another opcode.
I'm not saying it's impossible to fold the re-binding into a (set of)
special new opcode(s), but I am saying it's more complex than at first
it appears.
FWIW, I spent some time about a year ago looking at things like this
(small improvements to the peephole optimizer which allowed certain very
common sequences to be folded into a (new) opcode which in turn allowed
other optimizations to avoid branching). The changes worked, but didn't
actually improve performance significantly in my tests (which is why I
ended up not bothering to propose anything).
I remember back in the day (circa 1.5.2?) that
trips-around-the-interpreter-loop were significant and avoiding them
could give wins. However, in the current CPython interpreter, the
improvements over the original huge switch() to dispatch the bytecodes
to the correct handler appear to have made this type of optimization
less effective. That was my conclusion at the time, anyway - I only had
about a week to experiment with it.
E.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Christian Gollwitzer writes:
> Am 18.04.17 um 08:21 schrieb Chris Angelico:
>> On Tue, Apr 18, 2017 at 4:06 PM, Christian Gollwitzer
>> wrote:
>>> Am 18.04.17 um 02:18 schrieb Ben Bacarisse:
>>>
Thanks (and to Grant). IO seems to be the canonical example. Where
some languages would force one to write
c = sys.stdin.read(1)
while c == ' ':
c = sys.stdin.read(1)
>>>
>>> repeat
>>> c = sys.stdin.read(1)
>>> until c != ' '
>>
>> Except that there's processing code after it.
>>
>
> Sorry, I misread it then - Ben's code did NOT have it, it looks like a
> "skip the whitespace" loop.
It also reads the first character that is not whitespace, so it's not
usable to *merely* skip the whitespace.
>> while True:
>> c = sys.stdin.read(1)
>> if not c: break
>> if c.isprintable(): text += c
>> elif c == "\x08": text = text[:-1]
>> # etc
>>
>> Can you write _that_ as a do-while?
>
> No. This case OTOH looks like an iteration to me and it would be most
> logical to write
>
> for c in sys.stdin:
> if c.isprintable(): text += c
> elif c == "\x08": text = text[:-1]
> # etc
>
> except that this iterates over lines. Is there an analogous iterator
> for chars? For "lines" terminated by something else than "\n"?
> "for c in get_chars(sys.stdin)" and
> "for c in get_string(sys.stdin, terminate=':')" would be nicely
> readable IMHO. Or AWK-like processing:
>
> for fields in get_fields(open('/etc/passwd'), RS='\n', FS=':'):
> if fields[2]=='0':
> print 'Super-User found:', fields[0]
I don't know if those exist in some standard library, but they are easy
to write, and I do it all the time. I don't need the chars one, but I do
tab-separated fields and line-separated groups of tab-separated fields,
and variations.
for s, sentence in enumerate(sentences(sys.stdin)):
for k, token in enumerate(sentence):
...
token[LEMMA] or warn('empty LEMMA', s, k, sentence)
...
The wrapper function around sys.stdin or other text source is different
depending on the data format. Sometimes it's messy, sometimes not. Any
messy details are hidden the wrapper.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Am 18.04.17 um 08:21 schrieb Chris Angelico:
On Tue, Apr 18, 2017 at 4:06 PM, Christian Gollwitzer wrote:
Am 18.04.17 um 02:18 schrieb Ben Bacarisse:
Thanks (and to Grant). IO seems to be the canonical example. Where
some languages would force one to write
c = sys.stdin.read(1)
while c == ' ':
c = sys.stdin.read(1)
repeat
c = sys.stdin.read(1)
until c != ' '
Except that there's processing code after it.
Sorry, I misread it then - Ben's code did NOT have it, it looks like a
"skip the whitespace" loop.
while True:
c = sys.stdin.read(1)
if not c: break
if c.isprintable(): text += c
elif c == "\x08": text = text[:-1]
# etc
Can you write _that_ as a do-while?
No. This case OTOH looks like an iteration to me and it would be most
logical to write
for c in sys.stdin:
if c.isprintable(): text += c
elif c == "\x08": text = text[:-1]
# etc
except that this iterates over lines. Is there an analogous iterator for
chars? For "lines" terminated by something else than "\n"?
"for c in get_chars(sys.stdin)" and
"for c in get_string(sys.stdin, terminate=':')" would be nicely readable
IMHO. Or AWK-like processing:
for fields in get_fields(open('/etc/passwd'), RS='\n', FS=':'):
if fields[2]=='0':
print 'Super-User found:', fields[0]
Christian
--
https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Christian Gollwitzer : > Am 18.04.17 um 02:18 schrieb Ben Bacarisse: >> Python opts for >> >> while True: >> c = sys.stdin.read(1) >> if c != ' ': break > > This loop would be the archetypical do..while or repeat...until to me. > > do > c = sys.stdin.read(1) > while c== ' ' No, the code continues. You want to do something with c, right? > is the most clear to me - and in fact this "while True; do something; > break" thingy is just an idiom to fake a do..while loop in Python. C > does have it, for example, and it is way better like this than the > abuse of assignment and comma operator in the condition. I do use do ... while in my C code whenever a loop would end in a conditional break. I happens exceedingly rarely, though. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Tue, Apr 18, 2017 at 4:06 PM, Christian Gollwitzer wrote: > Am 18.04.17 um 02:18 schrieb Ben Bacarisse: > >> Thanks (and to Grant). IO seems to be the canonical example. Where >> some languages would force one to write >> >> c = sys.stdin.read(1) >> while c == ' ': >> c = sys.stdin.read(1) >> >> Python opts for >> >> while True: >> c = sys.stdin.read(1) >> if c != ' ': break > > > This loop would be the archetypical do..while or repeat...until to me. > > do > c = sys.stdin.read(1) > while c== ' ' > > > -or- > > repeat > c = sys.stdin.read(1) > until c != ' ' Except that there's processing code after it. while True: c = sys.stdin.read(1) if not c: break if c.isprintable(): text += c elif c == "\x08": text = text[:-1] # etc Can you write _that_ as a do-while? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Am 18.04.17 um 02:18 schrieb Ben Bacarisse: Thanks (and to Grant). IO seems to be the canonical example. Where some languages would force one to write c = sys.stdin.read(1) while c == ' ': c = sys.stdin.read(1) Python opts for while True: c = sys.stdin.read(1) if c != ' ': break This loop would be the archetypical do..while or repeat...until to me. do c = sys.stdin.read(1) while c== ' ' -or- repeat c = sys.stdin.read(1) until c != ' ' is the most clear to me - and in fact this "while True; do something; break" thingy is just an idiom to fake a do..while loop in Python. C does have it, for example, and it is way better like this than the abuse of assignment and comma operator in the condition. Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
Steve D'Aprano wrote: I'm not sure why the Cython devs maintain this is not a JIT compiler. Perhaps I misunderstand something. A JIT compiler works by observing the actual values taken on by variables at run time, and if it notices that a particular variable seems to always have a particular type, it compiles a special version of the code to handle that type, protected by appropriate guards. Cython doesn't do any of that -- it's just a plain, boring, standard ahead-of-time compiler that goes by the type info you give it and nothing more. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Tuesday, April 18, 2017 at 2:09:19 AM UTC+1, Paul Rubin wrote: > Ben Bacarisse writes: > > ? I get "AttributeError: 'itertools.dropwhile' object has no attribute > > 'next'" from your example. > > Hmm, .next() worked ok for me in Python 2.7.5. Not sure what happened. > Maybe something went wrong with my paste. Oh well. > PEP 3114 -- Renaming iterator.next() to iterator.__next__() https://www.python.org/dev/peps/pep-3114/ Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Ben Bacarisse : > Python opts for > > while True: > c = sys.stdin.read(1) > if c != ' ': break I opt for that in C and bash as well. In fact, when I start writing a loop, I first type: while True: Once it is done, I might notice that the loop begins: while True: if ... break and merge "while" and "if". Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Mon, 17 Apr 2017 08:52 pm, Steve D'Aprano wrote: > but research does continue into using gradual typing for optimizations: > > http://dl.acm.org/citation.cfm?id=2069175 Another interesting research project into speeding up Jython using type hints: http://scholar.colorado.edu/ecen_gradetds/57/ and more about gradual typing in Python: http://homes.soic.indiana.edu/mvitouse/talks/stop12.pdf -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Gregory Ewing : > Marko Rauhamaa wrote: >> What I notice in my numbers is that about one half of my while loops >> are "while True", and about a third of my loops are while loops. > > Out of curiosity, what proportion of your 'while True' loops are > infinite? (I.e. no break, return or raise in the loop.) 0% Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Tue, Apr 18, 2017 at 1:37 AM, MRAB wrote: > In Python 3 it's: > > c = next(itertools.dropwhile( > lambda c: c==' ', > iter(lambda: sys.stdin.read(1),None) > )) iter's sentinel should be an empty string. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 2017-04-18 02:09, Paul Rubin wrote: Ben Bacarisse writes: ? I get "AttributeError: 'itertools.dropwhile' object has no attribute 'next'" from your example. Hmm, .next() worked ok for me in Python 2.7.5. Not sure what happened. Maybe something went wrong with my paste. Oh well. Coming from the lazy language Haskell, I find your example natural... Yep ;) Your mistake was that you didn't say it was Python 2. :-) In Python 3 it's: c = next(itertools.dropwhile( lambda c: c==' ', iter(lambda: sys.stdin.read(1),None) )) -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Ben Bacarisse writes: > ? I get "AttributeError: 'itertools.dropwhile' object has no attribute > 'next'" from your example. Hmm, .next() worked ok for me in Python 2.7.5. Not sure what happened. Maybe something went wrong with my paste. Oh well. > Coming from the lazy language Haskell, I find your example natural... Yep ;) -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Paul Rubin writes: > Ben Bacarisse writes: >> c = sys.stdin.read(1) >> while c == ' ': >> c = sys.stdin.read(1) (for the record: I was not suggesting this was how you'd do it but how you'd be forced to do it in some languages) > c = itertools.dropwhile( > lambda c: c==' ', > iter(lambda: sys.stdin.read(1),None) > ).next() Did you mean c = next(itertools.dropwhile(lambda c: c==' ', iter(lambda: sys.stdin.read(1), None))) ? I get "AttributeError: 'itertools.dropwhile' object has no attribute 'next'" from your example. Coming from the lazy language Haskell, I find your example natural (though the syntax is a little busy for my taste) but would it be considered Pythonic? -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 18/04/2017 01:23, Paul Rubin wrote: Ben Bacarisse writes: c = sys.stdin.read(1) while c == ' ': c = sys.stdin.read(1) c = itertools.dropwhile( lambda c: c==' ', iter(lambda: sys.stdin.read(1),None) ).next() I tried this but it doesn't like the .next. I wanted to see if it was any faster or slower than the more obvious loop. (With the original loop, changing: c=sys.stdin.read(1) while c==' ': c=sys.stdin.read(1) to: r=sys.stdin.read c=r(1) while c==' ': c=r(1) made it 75% faster (reading 10M spaces). Those 3 extra lookups per loop I guess (this was run inside a function otherwise the dynamics are different). -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Marko Rauhamaa writes: > Ben Bacarisse : > >> Marko Rauhamaa writes: >>> What I notice in my numbers is that about one half of my while loops >>> are "while True", and about a third of my loops are while loops. >> >> I fo[u]nd the proportion on while True: loops surprising. Is there >> something about Python that encourages that kind of loop? (Thanks for th type correction. There is also s/of/on/.) > Here's a typical example of such a loop in Python (ver 2): > > while True: > try: > snippet = os.read(rdfd, 1000) > except OSError as e: > if e.errno == errno.EAGAIN: > return > raise > if not snippet: > break > self.stdout_snippets.append(snippet) Thanks (and to Grant). IO seems to be the canonical example. Where some languages would force one to write c = sys.stdin.read(1) while c == ' ': c = sys.stdin.read(1) and an Algol-68 style language would permit one to write while (c := read char; c) do skip od Python opts for while True: c = sys.stdin.read(1) if c != ' ': break (Forgive the atypical choice of input primitive -- it's for illustrating the loop style only.) -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Ben Bacarisse writes: > c = sys.stdin.read(1) > while c == ' ': > c = sys.stdin.read(1) c = itertools.dropwhile( lambda c: c==' ', iter(lambda: sys.stdin.read(1),None) ).next() -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Marko Rauhamaa wrote: What I notice in my numbers is that about one half of my while loops are "while True", and about a third of my loops are while loops. Out of curiosity, what proportion of your 'while True' loops are infinite? (I.e. no break, return or raise in the loop.) -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
Steve D'Aprano writes: > On the other hand, there's Cython. Cython claims *not* to be a JIT compiler, One of the uses of "JIT compiler" these days is what's sometimes called a tracing JIT, like PyPy or LuaJIT or the Javascript flavor-of-the-week. That means it interprets the program while collecting execution traces to figure out the actual types the program uses at runtime, then generates machine code for those specific code paths, with some guards in case an unexpectedly typed value shows up. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 2017-04-17, Ben Bacarisse wrote: > Marko Rauhamaa writes: > >> Terry Reedy : >> >>> On 4/17/2017 3:11 AM, Marko Rauhamaa wrote: Here's statistics from a medium-sized project of mine: while True:34 while : 39 for ... in ...: 158 >>> >>> As I posted previously, the ratio of for-loops in the stdlib is about 7 >>> to 1. >> >> What I notice in my numbers is that about one half of my while loops are >> "while True", and about a third of my loops are while loops. > > I fond the proportion on while True: loops surprising. Is there > something about Python that encourages that kind of loop? That's how you read data from a socket/pipe/file. -- Grant Edwards grant.b.edwardsYow! Your CHEEKS sit like at twin NECTARINES above gmail.coma MOUTH that knows no BOUNDS -- -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 17 April 2017 at 04:00, Steve D'Aprano wrote:
> On Mon, 17 Apr 2017 05:49 am, Dennis Lee Bieber wrote:
>
>> On Mon, 17 Apr 2017 02:48:08 +1000, Steve D'Aprano
>> declaimed the following:
>>
>>>On Sun, 16 Apr 2017 11:57 pm, bartc wrote:
>>>
But people just don't want it.
>>>
>>>Damn straight. Now you get it. It's not about how easy it is to implement,
>>>it's about whether it is needed and wanted. It isn't needed, and it isn't
>>>wanted.
>>>
>>
>> Let's go all the way, and just get rid of "while ".
>
> Sure, why not? If it's your language, you can make it as minimalist or
> maximalist as you like. You get to choose.
>
I think it is right not to touch anything for .py file rules.
Seriously there are much more problems, and better code editors
should adress those, not current syntax.
As for loops, well, if we speak of some 'my' language,
it could be interesting whether there are common patterns.
But is it on-topic here to speak of, say PyBasic2020 fantasy
language?
For real-time application development the amount of
"while(condition)" vs "while True" can play bad jokes
with readability.
I would find such looping schema simpler:
loop:
if contitioin 1 : contitioin_break = 1
blabla bla = bla + foo
make something
make something
if contitioin 2 : contitioin_break = 1
if contitioin 3 : contitioin_break = 0
if contitioin_break : *break
/loop
Anyway it is just question where to show this last line,
or to make an emphasis on it.
so I'd just leave it there, for me it helps when I imagine
a cascade of functions and my eyes go down to the
bottom. If I have kilometers of "if"s inside the loop,
then hiding one line can become more a problem than an aid.
Constructs like "repeat 100 times {}" or other sort of
"let's pronounce it in English" found in
some languages looks sort of confusing, it takes
me time to "unwrap" their sense into a cascade
especially when there are many different keywords.
Mikhail
--
https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Ben Bacarisse : > Marko Rauhamaa writes: >> What I notice in my numbers is that about one half of my while loops >> are "while True", and about a third of my loops are while loops. > > I fo[u]nd the proportion on while True: loops surprising. Is there > something about Python that encourages that kind of loop? Here's a typical example of such a loop in Python (ver 2): while True: try: snippet = os.read(rdfd, 1000) except OSError as e: if e.errno == errno.EAGAIN: return raise if not snippet: break self.stdout_snippets.append(snippet) I find myself writing similar loops in C a lot, as well. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 17/04/2017 19:02, Ben Bacarisse wrote:
Marko Rauhamaa writes:
Terry Reedy :
On 4/17/2017 3:11 AM, Marko Rauhamaa wrote:
Here's statistics from a medium-sized project of mine:
while True:34
while : 39
for ... in ...: 158
As I posted previously, the ratio of for-loops in the stdlib is about 7
to 1.
What I notice in my numbers is that about one half of my while loops are
"while True", and about a third of my loops are while loops.
I fond the proportion on while True: loops surprising. Is there
something about Python that encourages that kind of loop?
A few things:
(1) Python doesn't have the equivalent of C's comma operator, and no
assignment in expressions. So if a condition relies on such set-up code,
it can't do this:
while (c=nextc())!=0:
But has to do this or use extra logic:
while 1:
c=nextc()
if c==0: break
(2) There are exceptions ('raise') as an extra means of breaking out of
such loops (as well as return, break, and exit()
(3) There's 'yield' too, as I've just seen this example:
while True:
yield []
So there are more excuses to make use of it. Plus Python programmers may
be more averse to using convoluted logic just to avoid a 'break' in the
middle of a loop.
--
bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Marko Rauhamaa writes: > Terry Reedy : > >> On 4/17/2017 3:11 AM, Marko Rauhamaa wrote: >>> Here's statistics from a medium-sized project of mine: >>> >>>while True:34 >>>while : 39 >>>for ... in ...: 158 >> >> As I posted previously, the ratio of for-loops in the stdlib is about 7 >> to 1. > > What I notice in my numbers is that about one half of my while loops are > "while True", and about a third of my loops are while loops. I fond the proportion on while True: loops surprising. Is there something about Python that encourages that kind of loop? -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Terry Reedy : > On 4/17/2017 3:11 AM, Marko Rauhamaa wrote: >> Here's statistics from a medium-sized project of mine: >> >>while True:34 >>while : 39 >>for ... in ...: 158 > > As I posted previously, the ratio of for-loops in the stdlib is about 7 > to 1. What I notice in my numbers is that about one half of my while loops are "while True", and about a third of my loops are while loops. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 4/17/2017 3:11 AM, Marko Rauhamaa wrote: Gregory Ewing : bartc wrote: Most of my loops start off as endless loops, until I can determine the actual terminating condition, and where it best goes. Interesting. My experience is quite different. Most of the loops I write start off with me thinking "Now I want to do this for each element of this collection", which maps straight to a Python for-loop. In the rare cases where I do use a while loop, usually I have a fairly good idea what the terminating condition is going to be. Here's statistics from a medium-sized project of mine: while True:34 while : 39 for ... in ...: 158 As I posted previously, the ratio of for-loops in the stdlib is about 7 to 1. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Mon, 17 Apr 2017 04:18 pm, Paul Rubin wrote: > Steve D'Aprano writes: >> Since it is *optional*, it is only a hint, not a fact. You can tell the >> compiler that you believe that n will be an int, but it's not guaranteed. > > The runtime could check at the entry to the function that n is an int, > and then it wouldn't have to keep re-checking on the inside of a loop. > That's what a JIT compiler does in the absence of annotations, more or > less; Funnily enough, in the very post you quote, I did write: "There are clever Just In Time compiler tricks that can optimize the code, which is what Psycho did and PyPy still does, but they don't need type hints." The reason being that at runtime the compiler knows the type of the value and the type hint is redundant. On the other hand, there's Cython. Cython claims *not* to be a JIT compiler, but I don't really understand why. As I understand it, Cython can operate in two ways: (1) as a Python interpreter, with some optimizations switched on or off at runtime according to the types and/or values actually used by the program; (2) as a pseudo-Python to C compiler, allowing you to write Python-ish code that is compiled to C (plus a Python runtime for anything that's too hard to translate into C). I'm not sure why the Cython devs maintain this is not a JIT compiler. Perhaps I misunderstand something. > but the annotations make life easier for ahead-of-time compilers. > Again this type of thing has been standard in Lisp since the 1960's. Indeed it has. The state of the art of optimizations for dynamically typed languages has been badly neglected for many years. Fortunately in the last decade or so there's been a lot more interest in this. See, for example, Steve Yegge: http://steve-yegge.blogspot.com.au/2008/05/dynamic-languages-strike-back.html -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Sun, 16 Apr 2017 02:27 pm, Steve D'Aprano wrote: > Are you aware that optional static typing CANNOT be used for optimization? I think I've over-stated that. Considerably. In other words, I was wrong. As Steve Yegge points out, dynamic languages like Smalltalk and Lisp were, back in the 1980s, heavily optimized and were competitive with statically-typed languages like C: http://steve-yegge.blogspot.com.au/2008/05/dynamic-languages-strike-back.html While this is true: > Since it is *optional*, it is only a hint, not a fact. You can tell the > compiler that you believe that n will be an int, but it's not guaranteed. > As the Dude says, the type hint is > > that's, like, only your opinion man > > and the compiler cannot trust it. It must still guard the operation with a > runtime check that n actually is an int, The conclusion that I drew: > and so you lose most or all of > the benefit of telling the compiler that it's an int. is not necessarily the case. If you're doing lots of little operations on disparate variables, with lots and lots of guard checks and not much optimized code, it will go badly for you, but on the other hand for big loops with only a few up-front checks and predictable types, there may be opportunities for optimization. Honestly, I'm not quite sure what I was thinking: that sort of "optimized code with a guard" is exactly what Psycho does, and what FATPython is hoping to bring to CPython some day. At the present time, the state of the art of gradual typing is still keeping the emphasis on optional static typing for *correctness*, not optimization: http://wphomes.soic.indiana.edu/jsiek/what-is-gradual-typing/ but research does continue into using gradual typing for optimizations: http://dl.acm.org/citation.cfm?id=2069175 [...] > Python's gradual typing is not about optimization, it is about testing > program correctness and helping to find bugs at compile time*. That's the motivation for MyPy. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Moderating the list [was: Python and the need for speed]
On Thursday, April 13, 2017 at 10:31:22 AM UTC+1, Tim Golden wrote: > On 13/04/2017 03:39, Jason Friedman wrote: > >> > >> However, it's simply a technical fact: the thing which we moderate is the > >>> mailing list. We can control which posts make it through from the > >>> newsgroup > >>> by blocking them at the gateway. But the posts will continue to appear on > >>> comp.lang.python which is, as the description says, unmoderated. > >>> > >> > >> TJG > > > > > > Thank you, Tim and Ethan and the other moderators, for performing that > > function. > > It makes the reading experience more pleasant for me. > > > > I looked, for the first time in a long while, at the GG mirror of the > newsgroup. Hadn't realised how much of that really nasty all-caps spam > we were blocking. (We set up some gateway rules when that started to > become a problem so we just don't see it on the mailing list). > > Most of the time we're just binning run-of-the-mill spam. Doesn't take > too much effort if you keep on top of it, but it piles up quickly enough > if you leave it for a bit! > > TJG I routinely set all of the upper-case crap to "violent content" on GG just as a matter of principle. There's even been a couple of English versions lately but the filters have stopped them, well done :-) Kindest regards. Mark Lawrence. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Gregory Ewing : > bartc wrote: >> Most of my loops start off as endless loops, until I can determine >> the actual terminating condition, and where it best goes. > > Interesting. My experience is quite different. Most of the loops I > write start off with me thinking "Now I want to do this for each > element of this collection", which maps straight to a Python for-loop. > > In the rare cases where I do use a while loop, usually I have a fairly > good idea what the terminating condition is going to be. Here's statistics from a medium-sized project of mine: while True:34 while : 39 for ... in ...: 158 Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
Steve D'Aprano writes: > Since it is *optional*, it is only a hint, not a fact. You can tell the > compiler that you believe that n will be an int, but it's not guaranteed. The runtime could check at the entry to the function that n is an int, and then it wouldn't have to keep re-checking on the inside of a loop. That's what a JIT compiler does in the absence of annotations, more or less; but the annotations make life easier for ahead-of-time compilers. Again this type of thing has been standard in Lisp since the 1960's. -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Mon, 17 Apr 2017 05:16 am, bartc wrote: > But it was OK for Steve to 'win' the benchmark by substituting my test > code with something only vaguely related, and much simpler? Okay, you're now being obnoxious, and telling lies about me. What I said was "FOR WHAT IT'S WORTH [emphasis added], on my machine (2GB RAM and 1.2GHz CPU) I can add up 100 million ints in 14 seconds" which is just to give an indication of the speed of my machine. I wasn't trying to "win" your stupid, inefficient benchmark. If I was, would I then have run *your code* and report a time nearly three times longer than you? "time taken: 93.543194 seconds" How is this me winning by running different code? You also ignored the entire point of my post, which is to comment on Justin's generator version: "If I can run Bart's test on my slow old computer in a minute and a half, my guess is that your generator must have been doing something terribly wrong to still not be complete after six minutes." and misrepresent me as doing something underhanded and dishonorable to "win". And that, Bart, is dirty pool. I don't think you're arguing in good faith. I've spent months defending your right to criticise Python on this forum even when others have wanted you banned (an excessive reaction) or have kill-filed you so they don't see your posts. Even when I disagree with your conclusions, even if I think they are terribly naive, the sorts of things a lone cowboy developer can get away with but not suitable for a mature, widely used language like Python, I've respected your point of view. But at this point, I think *you* are being dishonest. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Mon, 17 Apr 2017 05:49 am, Dennis Lee Bieber wrote: > On Mon, 17 Apr 2017 02:48:08 +1000, Steve D'Aprano > declaimed the following: > >>On Sun, 16 Apr 2017 11:57 pm, bartc wrote: >> >>> But people just don't want it. >> >>Damn straight. Now you get it. It's not about how easy it is to implement, >>it's about whether it is needed and wanted. It isn't needed, and it isn't >>wanted. >> > > Let's go all the way, and just get rid of "while ". Sure, why not? If it's your language, you can make it as minimalist or maximalist as you like. You get to choose. Python's not your language, so it's not your choice for Python. But you can always create your own language and implement as many or as few features as you like. Guido likes while loops with a condition, and Python is Guido's language. And of course, since Python has had while loops with a condition for over 20 years now, backwards compatibility precludes removing it. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
bartc wrote: > - describing the various syntax forms; > - explaining how they differ; > - tutorials for beginners showing each form; And you don't have to explain how an endless loop should be written as 'while True', meanwhile advising against using 'while 1'? You don't have to mention it in the reference docs, because it's deducible from the rest of the language rules. Nor do you need an explicit test for it, unless there is code in the compiler specifically to optimise that case. Tutorials might want to mention it, but it's not strictly necessary, as a smart enough student could figure it out for themselves. So, how many kinds of sequences do you have in Python? lists tuples namedtuples arrays bytearrays string ? memoryview By far the most commonly used ones are list and tuple. The rest are designed for specialised uses, and have characteristics that can't easily be replicated using the fundamental types, if at all. Alternative looping constructs, in contrast, have *no* functional difference, their only advantage being that they provide a slightly more succint way of expressing certain things. While a; b; c; d Do e; f; g End with the test (the value of 'd') effectively in the middle. That would be a rather obtuse way of expressing it, though. It looks like the whole of 'a; b; c; d' is the condition, whereas you're really only using a, b and c for their side effects. The whole point of a loop-and-a-half construct is to make the condition stand out, despite being in the middle. This is actually the only additional looping construct that I wouldn't mind seeing. It gets brought up now and then; last time, if I recall, the discussion fizzled out due to a feeling that you don't really need it much in Python, because those kinds of loops are usually better expressed using an iterator. Most of my loops start off as endless loops, until I can determine the actual terminating condition, and where it best goes. Interesting. My experience is quite different. Most of the loops I write start off with me thinking "Now I want to do this for each element of this collection", which maps straight to a Python for-loop. In the rare cases where I do use a while loop, usually I have a fairly good idea what the terminating condition is going to be. Cases where a significant amount of head scratching is needed to figure it out are pretty rare. But maybe you're working on different kinds of problems from me. Sometimes they stay as endless loops. By design or by accident? :-) Every program I've every written that had a deliberate endless loop in it had exactly *one* of them, at the top level. And I knew it would be that way before I started writing it. (And since the only way to stop it would be ctrl-C or equivalent, I only write such things if they're throwaway tools for my own use.) -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 16/04/2017 19:42, Chris Angelico wrote:
On Mon, Apr 17, 2017 at 4:21 AM, bartc wrote:
Here is a function from some old CPython source that appears to be something
to do with While statements:
static int
validate_while(node *tree)
{
...
Look, no comments! Are you going to castigate the developers of CPython for
that? (I guess not.)
Can you find the equivalent in the current sources? I think it might be here:
https://github.com/python/cpython/blob/master/Grammar/Grammar#L73
while_stmt: 'while' test ':' suite ['else' ':' suite]
A single, extremely readable line. Abstract, not concrete.
How do you know my code fragment wasn't also generated? Since it was
parsing C, the refs for that language also define the while statement
quite neatly:
'while' '(' expression ')' statement
but actually it was done manually. It's part of a suite of functions
that do things in a common pattern, and would be documented as a set
rather than line by line. And elsewhere calls to such a function would
also be part of a chain of tests.
That's if I needed to documented it (but I can't remember how many
parsers I've done along the same lines since c.1979).
But I've glanced through the CPython code again, and it is quite lightly
commented. Although sometimes it gets quite chatty.
Want to show me your source for this source?
The source files I looked at were dated around 2010. I've no idea where
they came from. I think I'd wanted to compile it, but unfortunately,
they're not written in C, but a combination of C, make-file,
compiler-options, and bash-script with a bunch of Linux dependencies.
Next complaint, please.
No complaint, just saying other projects are sparse on comments too. And
my code isn't intended as open source.
--
bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Mon, Apr 17, 2017 at 5:16 AM, bartc wrote: > On 16/04/2017 20:00, Chris Angelico wrote: >> >> On Mon, Apr 17, 2017 at 4:43 AM, bartc wrote: > > >> Sure! If all you care about is winning benchmarks, > > > The benchmarks should be about comparing things. But they have to be like > for like. You're the one who suggested using a formula to calculate the sum of a sequence of integers. > But it was OK for Steve to 'win' the benchmark by substituting my test > code with something only vaguely related, and much simpler? Steven was, far as I can tell, reimplementing the same algorithm in more Pythonic code. But I'm not sure. I didn't look at the details. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On 16/04/2017 20:00, Chris Angelico wrote: On Mon, Apr 17, 2017 at 4:43 AM, bartc wrote: Sure! If all you care about is winning benchmarks, The benchmarks should be about comparing things. But they have to be like for like. Since this was about the effects of type annotations, it would be interesting to see how they would affect timings in the Python example. (I showed the affect on another language, where I got a factor of about 100:1 between slowest and fastest by using type declarations. A bit of a cheat though as the typed language was really a separate one.) you could easily create a sum() that recognizes when it's given a range object. Let's start with a baseline. I took the code from the link Steven posted (and renamed Timer to Stopwatch), and: 49995000 time taken: 1.143872 seconds Ehh, too fast. It is fast. My Python 3.4 on Intel 3.2GHz took 6 seconds. Even pypy took over 4 seconds. time taken: 0.16 seconds Hey look! We win at benchmarks. But it was OK for Steve to 'win' the benchmark by substituting my test code with something only vaguely related, and much simpler? -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Mon, Apr 17, 2017 at 4:43 AM, bartc wrote: >> >> For what it's worth, on my machine (2GB RAM and 1.2GHz CPU) I can add up >> 100 >> million ints in 14 seconds: >> >> py> with Stopwatch(): >> ... sum(range(1)) >> ... >> 49995000 >> time taken: 14.032116 seconds > > > I'm surprised it took that long. If a 'range' is no longer expanded to a > list, and 'sum' is built-in, isn't there a formula that can be used to sum a > sequence? At least when the step is +1. Sure! If all you care about is winning benchmarks, you could easily create a sum() that recognizes when it's given a range object. Let's start with a baseline. I took the code from the link Steven posted (and renamed Timer to Stopwatch), and: 49995000 time taken: 1.143872 seconds Ehh, too fast. I'll add another zero onto it. 45 time taken: 11.413530 seconds Better. Now then. _orig_sum = sum def sum(iterable, start=0): if isinstance(iterable, range) and iterable.step == 1: # I don't feel like dealing with non-unity steps n = iterable.stop * (iterable.stop - 1) // 2 if iterable.start: n -= iterable.start * (iterable.start + 1) // 2 return start + n return _orig_sum(iterable, start) 45 time taken: 0.16 seconds Hey look! We win at benchmarks. Of course, to do this, I had to make every other sum() call slightly slower (maybe not a lot, but slower by the price of one instance check at best), but what a benefit! ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On 16/04/2017 18:13, Steve D'Aprano wrote: On Mon, 17 Apr 2017 02:20 am, justin walters wrote: On Sun, Apr 16, 2017 at 8:46 AM, bartc wrote: What were the results with Python on your machine? Well, at first I tried to implement it with a generator. I gave up on waiting for the program to complete after about 6 minutes. That seems... excessive. For what it's worth, on my machine (2GB RAM and 1.2GHz CPU) I can add up 100 million ints in 14 seconds: py> with Stopwatch(): ... sum(range(1)) ... 49995000 time taken: 14.032116 seconds I'm surprised it took that long. If a 'range' is no longer expanded to a list, and 'sum' is built-in, isn't there a formula that can be used to sum a sequence? At least when the step is +1. factor of ten between friends? Of course the built-in sum() is a bit faster than the version Bart uses. It's almost like he is deliberately writing the least efficient Python code he can... (or maybe he just doesn't know Python very well). Yes, it was low-level code, which is a large part of what optimising a language is about. You can't always write just high-level code that calls fast built-in routines; sometimes the language has to do some actual work of its own! And the same algorithm (if you can call it that) was used in all the examples. That it wasn't doing anything meaningful is not important; it is testing the same mechanisms that would be involved in a real program. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Mon, Apr 17, 2017 at 4:21 AM, bartc wrote:
> Here is a function from some old CPython source that appears to be something
> to do with While statements:
>
> static int
> validate_while(node *tree)
> {
> int nch = NCH(tree);
> int res = (validate_ntype(tree, while_stmt)
>&& ((nch == 4) || (nch == 7))
>&& validate_name(CHILD(tree, 0), "while")
>&& validate_test(CHILD(tree, 1))
>&& validate_colon(CHILD(tree, 2))
>&& validate_suite(CHILD(tree, 3)));
>
> if (res && (nch == 7))
> res = (validate_name(CHILD(tree, 4), "else")
>&& validate_colon(CHILD(tree, 5))
>&& validate_suite(CHILD(tree, 6)));
>
> return (res);
> }
>
> Look, no comments! Are you going to castigate the developers of CPython for
> that? (I guess not.)
Can you find the equivalent in the current sources? I think it might be here:
https://github.com/python/cpython/blob/master/Grammar/Grammar#L73
while_stmt: 'while' test ':' suite ['else' ':' suite]
A single, extremely readable line. Abstract, not concrete. And my
suspicion is that the code you're looking at above just might have
been generated from the same kind of grammar file - which means that
it is NOT the "CPython source", but rather an intermediate file. It's
the equivalent of looking at a .pyc file and complaining that it has
no comments.
Want to show me your source for this source? The best I can find (by
searching the git history (which was imported from the hg history) for
'validate_while') is the code prior to this commit:
https://github.com/python/cpython/commit/53595c
And I rather suspect, from some of the other comments in the area,
that this file may have been autogenerated. Even if it wasn't, it was
still not the "primary source", but exists merely to implement
Grammar/Grammar, as linked above.
Next complaint, please.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 16/04/2017 17:30, Steve D'Aprano wrote:
On Sun, 16 Apr 2017 10:06 pm, bartc wrote:
(The 30 Loc figure is with support for loops /in
general/ already in place, and is for /adding/ a new loop statement, in
this case 'while')
What part of *testing* and *documenting* do you not understand?
Do you have any unit tests for your compiler? How about regression tests --
when you fix a bug, do you write a regression test to ensure it never
creeps back in?
Do you have any documentation for your compiler? Does it include doctests?
Are there any tutorials that beginners can read?
I'm guessing you don't have any of those things. The code snippet you posted
doesn't even have any comments.
Here is a function from some old CPython source that appears to be
something to do with While statements:
static int
validate_while(node *tree)
{
int nch = NCH(tree);
int res = (validate_ntype(tree, while_stmt)
&& ((nch == 4) || (nch == 7))
&& validate_name(CHILD(tree, 0), "while")
&& validate_test(CHILD(tree, 1))
&& validate_colon(CHILD(tree, 2))
&& validate_suite(CHILD(tree, 3)));
if (res && (nch == 7))
res = (validate_name(CHILD(tree, 4), "else")
&& validate_colon(CHILD(tree, 5))
&& validate_suite(CHILD(tree, 6)));
return (res);
}
Look, no comments! Are you going to castigate the developers of CPython
for that? (I guess not.)
Anyway I took out the one or two comments from my posted extract.
If Python added something like:
loop N times:
...
we would need *at least* the following:
- a test that `loop` was interpreted as a keyword;
- a test that `times` was interpreted as a keyword;
- a test that `loop 0 times` didn't execute the body at all;
- a test that `loop 1 times` executed the body exactly once;
- a test that `loop N times` executed the body exactly N times, for
a few different (and probably randomly selected) values of N;
- a test that the statement handled negative integers and floats correctly;
- a test that the statement handled non-numeric values correctly
(probably by raising an exception);
- a test that `continue` worked as expected inside this statement;
- a test that `break` worked as expected;
- a test that `return` worked as expected;
- a test that `raise` worked as expected;
- a test that signals will be caught correctly inside the statement;
and possibly more.
Well, I was talking more about just 'loop:', to repeat endlessly. Then
half of those tests disappear. See, a dedicated statement can make
things simpler.
The other half would be the same for loops generally as, after you get
to the compiler output (if I can't call it byte-code), then the chances
are that the original identity of the loop is lost.
Loop N times could be on top of While or For. Or it could have its own
implementation where it /knows/ the loop counter is an integer, and a
64-bit one. See, another optimisation opportunity.
BTW supporting a dedicated endless loop was only about 20 lines in
another project. Believe me, it is /nothing/. I wish other aspects were
just as trivial. It didn't even need a dedicated keyword.
Of course it's "nothing" for you, since by the evidence given you don't
bother with the hard parts. No comments, no communication with other
developers (except to gloat over how awesome you are and how stupid their
choices are), no documentation, no tests.
There are a couple of aspects I like of dynamic languages, which is the
informality and spontaneity.
The informality with Python seems to have disappeared (with all this
paperwork, and committees and applications to get more features added;
about as interesting as making a proposal for a new office block).
While the spontaneity seems to have suffered too if a feature takes
years to be accepted, even if it would only take an hour or two to add
(on a smaller project like mine).
At least, I get the benefits immediately. If I had to wait five years
(when I might be dead anyway) then I might as well wait forever.
The loops I use are categorised as:
* Endless
That's just a special case of "until some condition is true".
No, because there is no condition to test or think about. There should
not be one to specify. Python requires that a dummy condition is
provided to satisfy the language syntax (but internally it compiles to
an endless loop, as there there is no bureaucracy telling it what to do).
* N times
That's just a special case of "over an integer sequence".
No, because you don't care about what that sequence is, nor about having
to present it as named object to the body of the loop. Another
optimisation opportunity.
Neither does the user really want to be bothered with writing the name
of a variable. And providing a range() is overkill, as you just want a
count not a sequence.
* Until some condition is true
You missed one: do you check the condition be
Re: Looping [was Re: Python and the need for speed]
On Mon, 17 Apr 2017 03:00 am, Rustom Mody wrote: > BTW I regard Steven's long list of things that youve missed such as > regression tests, docs etc to be somewhat off the mark > To see that try this experiment: > Just add a feature to python that matters to you along with all these > requirements: docs, tests etc etc > Your chances of acceptance may go up marginally but most likely it will > still not be accepted. That's very true. It's not enough to have working code, tests and documentation to get a feature accepted. I didn't think I implied otherwise, but if it wasn't clear, I'm sorry. The feature first needs to have a reason (use-cases). It must be justified. Unless you are Guido, "because I like it" is not enough. When you create your own language, like Guido did with Python and Bart has with his mystery language, your own personal taste is all it takes to get features included. Everyone else has to justify their request, and have it accepted by the core developers. But even once a feature is accepted *in principle* it won't be actually added to the language until it has tests and documentation. That's the point I'm trying to get across to Bart. It doesn't matter how trivial the implementation, or now much of a no-brainer it is, it still needs tests and documentation. In the specific case of "loop forever", I'm pretty sure that it would really struggle to gain community favour. But even if it did, I'm pretty sure the core devs and Guido in particular would reject it. As I've said before, I actually like the syntax "repeat forever" -- in the right context. In Hyperscript, it suits the rest of the language and reads well. In Python, well, it is like grafting the head of a lion onto the shoulder of a horse. Lions are handsome beasts, and horses are also very attractive, individually, but making a freakish chimera of the two is not. There are lots of ways to design programming languages, and personally I'm particularly fond of Hyperscript's verbose English-like syntax, Forth and other RPN stack-based languages, and Python. That's three very different styles, and I like them all. But not all together at once. Bart, if your aesthetic taste says different, that's your right, and good on you. Nobody wants to tell you what syntactic features you should or shouldn't add to your language. I just wish you would return the favour to the Python community and stop insisting that *your* taste is the only good taste. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Mon, 17 Apr 2017 02:20 am, justin walters wrote: > On Sun, Apr 16, 2017 at 8:46 AM, bartc wrote: > >> What were the results with Python on your machine? > > > > Well, at first I tried to implement it with a generator. I gave up on > waiting for the program to complete > after about 6 minutes. That seems... excessive. For what it's worth, on my machine (2GB RAM and 1.2GHz CPU) I can add up 100 million ints in 14 seconds: py> with Stopwatch(): ... sum(range(1)) ... 49995000 time taken: 14.032116 seconds which isn't fast but it's within an order of magnitude of fast. What's a factor of ten between friends? Of course the built-in sum() is a bit faster than the version Bart uses. It's almost like he is deliberately writing the least efficient Python code he can... (or maybe he just doesn't know Python very well). Here's his version again (slightly adapted): def add(a, b): return a + b def testfn(): total = a = b = 0 for i in range(1): total += add(a, b) a += 1 b += 2 return (a, b, total) And the results: py> with Stopwatch(): ... testfn() ... (1, 2, 149985000) time taken: 93.543194 seconds If I can run Bart's test on my slow old computer in a minute and a half, my guess is that your generator must have been doing something terribly wrong to still not be complete after six minutes. The timer I used (Stopwatch) is basically this: http://code.activestate.com/recipes/577896-benchmark-code-with-the-with-statement/ > After using your code almost exactly I get: Sum: 149985000 ***Execution Time: 17.33912420272827 *** > > That's with Python 3.5 on an Intel I5-6600k 3.9Ghz cpu and 16GB DDR4 > memory. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 4/16/2017 11:35 AM, Michael Torrie wrote: On 04/16/2017 07:57 AM, bartc wrote: But people just don't want it. /That/ is what surprises me, when people reject things that to me are no-brainers. Whereas to me, it is a no-brainer that we are better off *without* multiple while/loop constructs. I simply don't care about these missing loop constructs. I do ;-) I consider the current simplicity a feature. > Python works great for what I use it for, and apparently works well for many people. The great majority* of 'repetition with variation' is sequentially processing items from a collection. Python does that nicely with 'for item in collection: process(item)'. While-loops take care of everthing else. *I grepped 3.6.1 .../lib/*.py with the REs '^ *while ' and '^ *for ', recursing into subdirectories, including tests and a few packages in site-packages. These got 1389 and 9842 hits respectively. I am opposed to adding syntax to subdivide the 12% of looping using 'while'. (As a note, '^ *while True' had 363 hits, about 1/4 of while loops.) > I have yet to find a loop that I couldn't construct with Python's > apparently-limited constructs. For-loops, 'limited' to iterating through collections (iterables), cover at least 80% of cases. While-loops + continue/break easily cover the remaining cases of linear repetition. Branching repetition, as in naive Fibonacci calculation and tree processing, is more easily done with recursion. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Sunday, April 16, 2017 at 7:27:49 PM UTC+5:30, bartc wrote: > Technically, adding this one feature to Python /is/ trivial, ^ You are not paying attention bart and I am not likely to pursue this beyond this post. I tried to say as are others that the substantive reasons to reject a feature are mostly non-technical BTW I regard Steven's long list of things that youve missed such as regression tests, docs etc to be somewhat off the mark To see that try this experiment: Just add a feature to python that matters to you along with all these requirements: docs, tests etc etc Your chances of acceptance may go up marginally but most likely it will still not be accepted. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Sun, 16 Apr 2017 11:57 pm, bartc wrote: > Yet countless other, far more elaborate features /are/ added all the time. Indeed. Because they are needed. Because they add functionality that Python doesn't already have, or seriously improves the interface to that functionality. > Technically, adding this one feature to Python /is/ trivial, for > example, allowing while: as a synonym for while True:, but preferably > using a new keyword such as loop. "Preferably using a new keyword"? Dear gods, you really have *no clue* about the responsibilities of maintaining a programming language used by tens of thousands of people. Adding your new keyword will break thousands of peoples programs. That's not to be done lightly, on a whim, just because you don't like the spelling `while True`. py> loop = True File "", line 1 loop = True ^ SyntaxError: invalid syntax > Nothing else needs to be touched. Right. We don't need to test this new feature, because we have such awesome mad skillz that of course it will work perfectly first time. Nor do we need to document this new feature. We don't need to include it in the "What's New" documentation, or include it in the tutorial, or mention it in the list of keywords, or in the description of the grammar. Documentation is for people who don't already know the source code because they wrote it. People will work it out for themselves. Eventually. After many hours, or days, of pulling their hair out at why their working code has suddenly stopped working after upgrading to a new Python version. The only thing you could do that would be more irresponsible would be to insist on introducing this new, undocumented, untested, unnecessary feature in a bug-fix point release. Ah, but I'm guessing that you don't understand the need for point releases either? With only one user, yourself, who needs releases or version numbers? > And > it could have been done right at the start when the politics was simpler. > > But people just don't want it. Damn straight. Now you get it. It's not about how easy it is to implement, it's about whether it is needed and wanted. It isn't needed, and it isn't wanted. > /That/ is what surprises me, when people reject things that to me are > no-brainers. Well, the thing is, there are two kinds of no-brainers. There are the kind of things that are so obviously a good idea that it requires no brain to want it. And then there are the kinds of things that are so obviously a bad idea that it requires no brain to want it. Which is this, do you think? -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Sun, 16 Apr 2017 10:06 pm, bartc wrote: > On 16/04/2017 03:51, Steve D'Aprano wrote: >> On Sat, 15 Apr 2017 10:17 pm, bartc wrote: > >>> Yes, I'm constantly surprised at this, as such syntax has a very low >>> cost (in my last compiler, supporting 'while' for example only added 30 >>> lines to the project). >> >> That's the advantage of writing your own private language and having no >> users except for yourself. You get to cut corners. Python has tens of >> thousands of users, and doesn't have that luxury. > > Here are the lines of code in my C compiler which are necessary to > support 'while': https://pastebin.com/BYFV7EWr > > (45 lines shown, but there are exactly 30 Loc if blanks are excluded.) > > I'd be interested in knowing why implementing While in Python would need > significantly more. Have you looked at the source code? https://github.com/python/cpython > (The 30 Loc figure is with support for loops /in > general/ already in place, and is for /adding/ a new loop statement, in > this case 'while') What part of *testing* and *documenting* do you not understand? Do you have any unit tests for your compiler? How about regression tests -- when you fix a bug, do you write a regression test to ensure it never creeps back in? Do you have any documentation for your compiler? Does it include doctests? Are there any tutorials that beginners can read? I'm guessing you don't have any of those things. The code snippet you posted doesn't even have any comments. Here is the declaration and comment for a regression test in Python's test suite, checking against the return of a bug in the while statement: def test_break_continue_loop(self): """This test warrants an explanation. It is a test specifically for SF bugs #463359 and #462937. The bug is that a 'break' statement executed or exception raised inside a try/except inside a loop, *after* a continue statement has been executed in that loop, will cause the wrong number of arguments to be popped off the stack and the instruction pointer reset to a very small number (usually 0.) Because of this, the following test *must* written as a function, and the tracking vars *must* be function arguments with default values. Otherwise, the test will loop and loop. """ (I've reformatted the comment as a docstring to make it easier to wrap.) The test itself is 16 lines of code, plus 8 more lines of explanation for why the test exists. That's for just one bug. Here's a snippet of another test, testing that when "while 0" is optimized away, the else clause is not: # Issue1920: "while 0" is optimized away, # ensure that the "else" clause is still present. x = 0 while 0: x = 1 else: x = 2 self.assertEqual(x, 2) Do I need to go on? As a lone-wolf developer with a user base of exactly one person, perhaps you don't care about tests, documentations, tutorials, or even comments in your code. But Python has multiple dozens of developers who have to understand each others code. It is open source and receives patches and contributions from hundreds more. It has tens or hundreds of thousands of users with high expectations about quality. Python's requirements are a bit more strict than "well, it seems to be working okay, so that's good enough for me". If Python added something like: loop N times: ... we would need *at least* the following: - a test that `loop` was interpreted as a keyword; - a test that `times` was interpreted as a keyword; - a test that `loop 0 times` didn't execute the body at all; - a test that `loop 1 times` executed the body exactly once; - a test that `loop N times` executed the body exactly N times, for a few different (and probably randomly selected) values of N; - a test that the statement handled negative integers and floats correctly; - a test that the statement handled non-numeric values correctly (probably by raising an exception); - a test that `continue` worked as expected inside this statement; - a test that `break` worked as expected; - a test that `return` worked as expected; - a test that `raise` worked as expected; - a test that signals will be caught correctly inside the statement; and possibly more. [...] > No, we're talking about a loop. It must be just about the simplest thing > to implement in a language (compared with a type system, or code > generation). It still needs tests and documentation. In the absence of formal correctness proofs, any code not covered by tests should be considered buggy. > BTW supporting a dedicated endless loop was only about 20 lines in > another project. Believe me, it is /nothing/. I wish other aspects were > just as trivial. It didn't even need a dedicated keyword. Of course it's "nothing" for you, since by the evidence given you don't bother with the hard parts. No comments, no communication with other dev
Re: Static typing [was Re: Python and the need for speed]
On Sun, Apr 16, 2017 at 8:46 AM, bartc wrote: > What were the results with Python on your machine? Well, at first I tried to implement it with a generator. I gave up on waiting for the program to complete after about 6 minutes. After using your code almost exactly I get: >>> Sum: 149985000 >>> ***Execution Time: 17.33912420272827 *** That's with Python 3.5 on an Intel I5-6600k 3.9Ghz cpu and 16GB DDR4 memory. > (Nice-looking language, I hadn't heard of it.) Yeah, it's pretty neat. Some parts of Nim aren't really production ready yet, but I think it has a bright future. proc test() = > var > sum = 0 > a = 0 > b = 0 > for i in 0..<1: > > Loop iterations should be 100 million, if this is one less. Missing one out > won't affect the timing, but will give a different result if comparing > implementations to see if they do the same thing. > It's not one less. It will execute exactly 100 million times. Meaning it will execute when i is 99,999,999 but not when i is 100,000,000. This gives exactly 100,000,000 iterations including the iteration for i = 0. It could alternatively be written as: for i in 1..1: Didn't realize I had messed up the increment on var b with my Nim implementation! Fixed: === import times proc add(a, b: int): int = result = a + b proc test() = var sum = 0 a = 0 b = 0 for i in 0..<1: sum += add(a, b) a += 1 b += 2 echo "a: " & $a & " b: " & $b & "\n" echo "Sum: " & $sum when isMainModule: var t0 = cpuTime() test() var t1 = cpuTime() echo "***Execution Time: " & $(t1 - t0) & "***\n" === The above Nim code is a tad bit slower with the larger var b: >>> a: 1 b: 2 >>> Sum: 149985000 >>> ***Execution Time: 2.888533*** With Speed Optimization: >>> a: 1 b: 2 >>> Sum: 149985000 >>> ***Execution Time: 2.843629*** With size optimization: >>> a: 1 b: 2 >>> Sum: 149985000 >>> ***Execution Time: 2.844876*** Compiled with release flag: >>> a: 1 b: 2 >>> Sum: 149985000 >>> ***Execution Time: 2.842312*** > Hmm, the optimiser is similar to mine then! I'm pretty new when it comes to compilers. Been trying to learn more and build my own but progress is fairly slow. I'm not sure what `nimc` is doing under the hood, but I do know that it transpiles to C before running either `gcc` or `clang`. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 16/04/2017 15:22, Chris Angelico wrote:
On Sun, Apr 16, 2017 at 11:57 PM, bartc wrote:
Technically, adding this one feature to Python /is/ trivial, for example,
allowing while: as a synonym for while True:, but preferably using a new
keyword such as loop. Nothing else needs to be touched. And it could have
been done right at the start when the politics was simpler.
But people just don't want it.
/That/ is what surprises me, when people reject things that to me are
no-brainers.
Maybe it's not a no-brainer to them. Maybe you need to make the case that:
1) It's worth stopping people from using the word "loop" as a variable
Actually I think 'loop' could still be used as a variable, since 'name:'
wouldn't occur in any other contexts. But a dual purpose 'loop' wouldn't
be elegant.
2) It's worth documenting another type of loop
3) It's worth having everyone need to know how to read another type of loop
Compared to all the other stuff in Python, seriously? Lambdas,
listcomps, generators, iterators, decorators, subclassing, class and
instance attributes, ... no problemo.
But faced with:
loop: print ("Hi!")
will lead to head-scratching?!
What's the benefit, aside from avoiding discussions like this?
Imagine if the discussion was instead about introducing a negate
operator so that you could write:
a = -b
instead of having to write:
a = 0 - b
It wouldn't be a big deal having to do the latter, but it's nicer not to.
I just find writing 'while 1' (or while (1) or for(;;) elsewhere) an
annoyance.
--
bartc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On 16/04/2017 16:00, justin walters wrote: On Sun, Apr 16, 2017 at 3:55 AM, bartc wrote: Example C of the same silly program in Python: def add(a,b): return a+b def testfn(): sum=a=b=0 for i in range(1): sum += add(a,b) a += 1 b += 2 print (a,b,sum) testfn() Timings: A (Pure HLL**) 13 seconds (dynamic/interpreted) A (With ASM module) 3 B (my compiler) 0.5 (static/compiled) C (Python 2/xrange) 30 C (Python 3)38 C (Pypy) 5 Just for fun I wanted to write this up in Nim to compare execution time. Nim has Python-esqe syntax but is statically (Nice-looking language, I hadn't heard of it.) typed and compiled. I think this is relevant to the discussion. Code looks like this: ``` import times proc add(a, b: int): int = result = a + b proc test() = var sum = 0 a = 0 b = 0 for i in 0..<1: Loop iterations should be 100 million, if this is one less. Missing one out won't affect the timing, but will give a different result if comparing implementations to see if they do the same thing. With 100M, the results are (of a,b and sum): 1 2 149985000 sum += add(a, b) a += 1 b += 1 b += 2, both to the ensure the same output, and in case there's a sneaky optimisation it can do for b+=1... echo "a: " & $a & " b: " & $b & "\n" echo "Sum: " & $sum when isMainModule: var t0 = cpuTime() test() var t1 = cpuTime() echo "***Execution Time: " & $(t1 - t0) & "***\n" ``` No optimization: ***Execution Time: 2.876923*** Optimized for speed: ***Execution Time: 2.844163*** Optimized for size: ***Execution Time: 2.844901*** Hmm, the optimiser is similar to mine then! Release option: ***Execution Time: 2.844021*** So, generally around 2.8 seconds. Not too bad for a GC'd language. There are probably some more optimizations I could make to improve execution time. What were the results with Python on your machine? -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 04/16/2017 07:57 AM, bartc wrote: > But people just don't want it. > > /That/ is what surprises me, when people reject things that to me are > no-brainers. I simply don't care about these missing loop constructs. Python works great for what I use it for, and apparently works well for many people. I have yet to find a loop that I couldn't construct with Python's apparently-limited constructs. I guess I don't see what having those extra looping constructs will give me in terms of productivity or even ease of use. -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Sun, Apr 16, 2017 at 3:55 AM, bartc wrote: > Example C of the same silly program in Python: > > def add(a,b): > return a+b > > def testfn(): > sum=a=b=0 > for i in range(1): > sum += add(a,b) > a += 1 > b += 2 > > print (a,b,sum) > > testfn() > > > Timings: > > A (Pure HLL**) 13 seconds (dynamic/interpreted) > A (With ASM module) 3 > A (With ASM module) 2(older version; hmmm...) > > B (my compiler) 0.5 (static/compiled) > B (via C/gcc-O3) 0.14 > > C (Python 2) 163 > C (Python 2/xrange) 30 > C (Python 3)38 > C (Pypy) 5 > Just for fun I wanted to write this up in Nim to compare execution time. Nim has Python-esqe syntax but is statically typed and compiled. I think this is relevant to the discussion. Code looks like this: ``` import times proc add(a, b: int): int = result = a + b proc test() = var sum = 0 a = 0 b = 0 for i in 0..<1: sum += add(a, b) a += 1 b += 1 echo "a: " & $a & " b: " & $b & "\n" echo "Sum: " & $sum when isMainModule: var t0 = cpuTime() test() var t1 = cpuTime() echo "***Execution Time: " & $(t1 - t0) & "***\n" ``` No optimization: ***Execution Time: 2.876923*** Optimized for speed: ***Execution Time: 2.844163*** Optimized for size: ***Execution Time: 2.844901*** Release option: ***Execution Time: 2.844021*** So, generally around 2.8 seconds. Not too bad for a GC'd language. There are probably some more optimizations I could make to improve execution time. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Sun, Apr 16, 2017 at 11:57 PM, bartc wrote: > Technically, adding this one feature to Python /is/ trivial, for example, > allowing while: as a synonym for while True:, but preferably using a new > keyword such as loop. Nothing else needs to be touched. And it could have > been done right at the start when the politics was simpler. > > But people just don't want it. > > /That/ is what surprises me, when people reject things that to me are > no-brainers. Maybe it's not a no-brainer to them. Maybe you need to make the case that: 1) It's worth stopping people from using the word "loop" as a variable 2) It's worth documenting another type of loop 3) It's worth having everyone need to know how to read another type of loop What's the benefit, aside from avoiding discussions like this? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 16/04/2017 13:22, Rustom Mody wrote: On Sunday, April 16, 2017 at 5:36:28 PM UTC+5:30, bartc wrote: On 16/04/2017 03:51, Steve D'Aprano wrote: On Sat, 15 Apr 2017 10:17 pm, bartc wrote: Yes, I'm constantly surprised at this, as such syntax has a very low cost (in my last compiler, supporting 'while' for example only added 30 lines to the project). That's the advantage of writing your own private language and having no users except for yourself. You get to cut corners. Python has tens of thousands of users, and doesn't have that luxury. Here are the lines of code in my C compiler which are necessary to support 'while': https://pastebin.com/BYFV7EWr (45 lines shown, but there are exactly 30 Loc if blanks are excluded.) I'd be interested in knowing why implementing While in Python would need significantly more. (The 30 Loc figure is with support for loops /in general/ already in place, and is for /adding/ a new loop statement, in this case 'while') You walk down a mountain path (trail in US-ese?) stopping here and there to smell a flower or spy a butterfly. Change to driving down a mountain road — quiet for the most part — is it ok to drive on the wrong side? Change to an expressway is it ok to change lanes randomly, not maintain speeds within upper (and lower!) limits? As others have tried to point out maintaining your own 1-man-language is one thing Maintaining a language - used by industry and academia - Used by FLOSS teams and commercial - by kids and old-hands - 1-man projects and large multiperson projects is quite another Dont get me wrong: I dont agree with everything about python. And the current(est) mess around async (and f-strings?) is (IMHO) going to cost more dearly than people expect... And we will have to wait another 10 years for that decision... My point is that you dont seem to have any estimate of the difference in momentum between your 'its-just-another-30-lines' language and a million user base language Yet countless other, far more elaborate features /are/ added all the time. Technically, adding this one feature to Python /is/ trivial, for example, allowing while: as a synonym for while True:, but preferably using a new keyword such as loop. Nothing else needs to be touched. And it could have been done right at the start when the politics was simpler. But people just don't want it. /That/ is what surprises me, when people reject things that to me are no-brainers. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Sun, Apr 16, 2017 at 8:55 PM, bartc wrote: > On 16/04/2017 05:27, Steve D'Aprano wrote: >> >> On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote: >> >>> apparently, the py-devs believe we >>> only deserve type declarations that do nothing to speed up >>> code execution (aka: type-hints), instead of type >>> declarations that could actually speed up the code. Go >>> figure! >>> >>> I'm not a fan of forced static typing, but i am a fan of >>> optional static typing. >> >> >> Are you aware that optional static typing CANNOT be used for optimization? >> >> Since it is *optional*, it is only a hint, not a fact. You can tell the >> compiler that you believe that n will be an int, but it's not guaranteed. > > > No, Rick distinguished between hints, and actual declarations. It doesn't > matter if the latter are optional. > > I played around with this at one time. All variables were of type 'variant', > unless they had an explicit type declaration. All of this is ignoring one very important point: subclassing. If I declare a variable "int", can it or can it not accept a subclass of int? Hint: neither answer is correct for the general case. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On Sunday, April 16, 2017 at 5:36:28 PM UTC+5:30, bartc wrote: > On 16/04/2017 03:51, Steve D'Aprano wrote: > > On Sat, 15 Apr 2017 10:17 pm, bartc wrote: > > >> Yes, I'm constantly surprised at this, as such syntax has a very low > >> cost (in my last compiler, supporting 'while' for example only added 30 > >> lines to the project). > > > > That's the advantage of writing your own private language and having no > > users except for yourself. You get to cut corners. Python has tens of > > thousands of users, and doesn't have that luxury. > > Here are the lines of code in my C compiler which are necessary to > support 'while': https://pastebin.com/BYFV7EWr > > (45 lines shown, but there are exactly 30 Loc if blanks are excluded.) > > I'd be interested in knowing why implementing While in Python would need > significantly more. (The 30 Loc figure is with support for loops /in > general/ already in place, and is for /adding/ a new loop statement, in > this case 'while') You walk down a mountain path (trail in US-ese?) stopping here and there to smell a flower or spy a butterfly. Change to driving down a mountain road — quiet for the most part — is it ok to drive on the wrong side? Change to an expressway is it ok to change lanes randomly, not maintain speeds within upper (and lower!) limits? As others have tried to point out maintaining your own 1-man-language is one thing Maintaining a language - used by industry and academia - Used by FLOSS teams and commercial - by kids and old-hands - 1-man projects and large multiperson projects is quite another Dont get me wrong: I dont agree with everything about python. And the current(est) mess around async (and f-strings?) is (IMHO) going to cost more dearly than people expect... And we will have to wait another 10 years for that decision... My point is that you dont seem to have any estimate of the difference in momentum between your 'its-just-another-30-lines' language and a million user base language -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
On 16/04/2017 03:51, Steve D'Aprano wrote: On Sat, 15 Apr 2017 10:17 pm, bartc wrote: Yes, I'm constantly surprised at this, as such syntax has a very low cost (in my last compiler, supporting 'while' for example only added 30 lines to the project). That's the advantage of writing your own private language and having no users except for yourself. You get to cut corners. Python has tens of thousands of users, and doesn't have that luxury. Here are the lines of code in my C compiler which are necessary to support 'while': https://pastebin.com/BYFV7EWr (45 lines shown, but there are exactly 30 Loc if blanks are excluded.) I'd be interested in knowing why implementing While in Python would need significantly more. (The 30 Loc figure is with support for loops /in general/ already in place, and is for /adding/ a new loop statement, in this case 'while') For language developers with responsibilities to users, the job doesn't stop at just implementing the feature in the language. It also needs to be tested and documented. As a general rule of thumb, every line of production code should expect to add at least ten lines of test code (unit tests, regression tests, integration tests, functional tests, doc tests, etc). So for your "30 lines" feature, that adds 300 lines of tests, and probably another page or two of documentation: No, we're talking about a loop. It must be just about the simplest thing to implement in a language (compared with a type system, or code generation). BTW supporting a dedicated endless loop was only about 20 lines in another project. Believe me, it is /nothing/. I wish other aspects were just as trivial. It didn't even need a dedicated keyword. > - describing the various syntax forms; > - explaining how they differ; > - tutorials for beginners showing each form; And you don't have to explain how an endless loop should be written as 'while True', meanwhile advising against using 'while 1'? The more choices you offer, the harder that decision becomes: - numeric Pascal or C-style loop - foreach style loop - repeat while condition (test at start) - repeat until condition (test at start) - do ... while condition (test at end) - do ... until condition (test at end) - repeat forever So, how many kinds of sequences do you have in Python? lists tuples namedtuples arrays bytearrays string ? memoryview plus all those ordered types. My head is already spinning! The loops I use are categorised as: * Endless * N times * Until some condition is true * Iterate over an integer sequence * Iterate over a set of values of some object Other languages like to have even more elaborate schemes. That includes advanced uses of Python's for loop, were, for example, there are multiple loop variables. I wonder how much testing that took to get it right? I don't remember the language, but I remember seeing one generalisation of the repeat/do loop that puts the test in the middle, rather than at the start or end of the loop. If I remember it was something like: DO setup code # executed once only REPEAT loop body # before the test WHILE condition # test loop body # after the test thus combining both the repeat while condition: ... and do: ... until condition styles of looping in one handy syntax. Perhaps you misremembered as that looks too unwieldy to be practical. If implemented orthogonally, then a loop like this: While A Do B End where both A and B can be any sequence of statements or expressions, would allow you to write: While a; b; c; d Do e; f; g End with the test (the value of 'd') effectively in the middle. But very common requirements are endless loops, and repeat N times without needing an explicit counter. If by "very common" you mean "occasionally", I agree. Most of my loops start off as endless loops, until I can determine the actual terminating condition, and where it best goes. Sometimes they stay as endless loops. (Sometimes, I turn a normal statement into an endless loop with a 'do' prefix. This is an unusual feature but I use it quite a bit: doswitch nextchar() # looping version of 'switch' when 'A'..'Z' then else exit # ie. break out of loop end ) Python's byte-code does at least optimise out the check that '1' is true, but that's not what the reader sees, which is 'loop while 1 is true'. And one day it will be: while l: body that can be mistaken for that common idiom. The problem there is not the loop, but the foolish use of lowercase l as a variable name. Maybe. But if 'while 1' wasn't used, then that's one less thing to double-check. (I just looked at a random bunch of Python code; there were 25 instances of 'while True:', but 40 of 'while 1:'.) -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Looping [was Re: Python and the need for speed]
Steve D'Aprano writes: > I don't remember the language, but I remember seeing one generalisation of > the repeat/do loop that puts the test in the middle, rather than at the > start or end of the loop. If I remember it was something like: > > DO > setup code # executed once only > REPEAT > loop body # before the test > WHILE condition # test > loop body # after the test > > thus combining both the > > repeat while condition: > ... > > and > > do: > ... > until condition > > styles of looping in one handy syntax. Was the setup code in the loop because that seems unnecessary (I know this is from memory)? I have a recollection of using something like do code before test while cond code after test od which has much the same effect. Algol 68 can also do this because the while condition can be a block (any expression can be a block in A68). The result was not to everyone's taste. > On Sat, 15 Apr 2017 10:17 pm, bartc wrote: >> Of course, it's possible to overdo it; if you look at Lisp, you'll lose >> yourself in the myriad looping options. > > "Myriad"? As I understand it, Lisp offers just *one* more loop construct > than the number you agree is the "minimum" needed: five. > > loop > loop for > do > dotimes > dolist I suppose it depends on how you count, but BartC did talk about options. Here are some of the loop options in Common Lisp: (loop x with ...) (loop for i in (...) by ...) (loop for i downfrom 10 above x by -2 ...) (loop for x = ... then (f x) ...) (loop across ...) (loop for s being symbols of ...) (loop for s being hash-keys in ... using ...) That's just the start. Then you can add "doing", "if" "when", "unless", "return", "collect", "sum[ming]", "maximizing" and others. Or, if you need them, you can have "initially", "finally", "repeat", "while", "until", "always", "never" or "thereis" parts, all the time losing (some of) those parts you don't need. -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On 16/04/2017 05:27, Steve D'Aprano wrote: On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote: apparently, the py-devs believe we only deserve type declarations that do nothing to speed up code execution (aka: type-hints), instead of type declarations that could actually speed up the code. Go figure! I'm not a fan of forced static typing, but i am a fan of optional static typing. Are you aware that optional static typing CANNOT be used for optimization? Since it is *optional*, it is only a hint, not a fact. You can tell the compiler that you believe that n will be an int, but it's not guaranteed. No, Rick distinguished between hints, and actual declarations. It doesn't matter if the latter are optional. I played around with this at one time. All variables were of type 'variant', unless they had an explicit type declaration. But it got complicated. I decided to keep the dynamic language pure, as after all I had a statically compiled language with exactly the same syntax if I needed it faster! Examples A and B here: https://pastebin.com/D89zP3LF Example C of the same silly program in Python: def add(a,b): return a+b def testfn(): sum=a=b=0 for i in range(1): sum += add(a,b) a += 1 b += 2 print (a,b,sum) testfn() Timings: A (Pure HLL**) 13 seconds (dynamic/interpreted) A (With ASM module) 3 A (With ASM module) 2(older version; hmmm...) B (my compiler) 0.5 (static/compiled) B (via C/gcc-O3) 0.14 C (Python 2) 163 C (Python 2/xrange) 30 C (Python 3)38 C (Pypy) 5 (Python 2 with 'range' locked up my machine for nearly 3 minutes because it presumably exhausted the 4GB memory just executing a simple loop. Have I mentioned that building a massive list just to execute 'for' was a stupid idea?) The pypy timing is not bad, however it is misleading: you can't for example deduce the execution time of one iteration by dividing by 100 million, as you can with the others (gcc-O3 excepted as I've no idea what it does). You only get that timing if you actually wanted 100 million iterations. (That doesn't necessarily mean that you have to use type declarations, like in C, Pascal and Java. A good modern compiler can infer types, like in ML and Haskell.) Yes a good, but more sophisticated, compiler. Unlike mine! (** The pure HLL timing might be reduced to ~ 9 seconds using 'label-pointers' for dispatching, but this limited portability when converted to C code, as not all C compilers support them. But it doesn't matter as I can use the ASM version for speed. I believe CPython also makes use of label-pointers when compiled with gcc - as happens on Linux - but not when compiled with MSVC, so that CPython on Windows is a little slower for that reason. I don't know if this is still the case. Those Python timings were on Windows...) -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Sun, 16 Apr 2017 09:48:15 +, alister wrote: > On Sun, 16 Apr 2017 14:27:28 +1000, Steve D'Aprano wrote: > >> On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote: >> >>> apparently, the py-devs believe we only deserve type declarations that >>> do nothing to speed up code execution (aka: type-hints), instead of >>> type declarations that could actually speed up the code. Go figure! >>> >>> I'm not a fan of forced static typing, but i am a fan of optional >>> static typing. >> >> Are you aware that optional static typing CANNOT be used for >> optimization? >> >> Since it is *optional*, it is only a hint, not a fact. You can tell the >> compiler that you believe that n will be an int, but it's not >> guaranteed. As the Dude says, the type hint is >> >> that's, like, only your opinion man >> >> and the compiler cannot trust it. It must still guard the operation >> with a runtime check that n actually is an int, and so you lose most or >> all of the benefit of telling the compiler that it's an int. >> >> There are clever Just In Time compiler tricks that can optimize the >> code, which is what Psycho did and PyPy still does, but they don't need >> type hints. Type hints are superfluous to a JIT compiler, since it >> already knows the runtime type information. But JIT compilers have >> their limitations, like the necessity of warm up time, increased >> complexity, and much heavier memory requirements. >> >> If you want to use type hints for the purposes of compile-time >> optimizations, then typing must be compulsory and enforced by the >> compiler. >> No longer optional hints, but mandatory statements of indisputable >> fact. >> >> (That doesn't necessarily mean that you have to use type declarations, >> like in C, Pascal and Java. A good modern compiler can infer types, >> like in ML and Haskell.) >> >> Python's gradual typing is not about optimization, it is about testing >> program correctness and helping to find bugs at compile time*. As such, >> the fact that type hints are optional and "only your opinion" doesn't >> matter. The promise made by the type checker is: >> >> IF what you say is true (the type hints are correct) >> THEN this code will be sound >> >> (or alternatively, it's unsound because you have a type mismatch, and >> get a compile-time error). As a fallback, you still have the runtime >> dynamic type checks. >> >> Unit and functional tests are great, and necessary. Just because your >> code compiles in C, Java or Pascal doesn't mean that it is correct. You >> still have to run the code and test that it does what you expect. >> >> "It compiles! Quick, ship it!" -- attributed to Microsoft developers >> >> Nevertheless, the problem with testing is that it can only prove the >> presence of bugs, not their absence. A test that fails proves that your >> code has at least one bug. A test that passes doesn't prove that there >> are no bugs. >> >> Static type checking helps to shrink the gap between "all of my tests >> pass, >> so there are no known bugs" and "there are no unknown bugs". And that's >> an important and powerful tool, especially for large and complex >> programs. >> >> Python's type-hints are for writing correct code, not fast code. I find >> it amusing when people argue that Python's type hints are pointless or >> useless. Do they realise they're effectively arguing that correct, >> bug-free code is pointless? >> >> "Your code is buggy. It calculates the wrong result." >> >> "Maybe so, but look how fast it does it!" > > I think you have (deliberately?) miss understood RR by optional Static > typing I think he means you don't have to use it as is in suing static > types is optional) but when you define a variable as static then its > type is fixed, not simply a hint. > > whether or not this could increase code deficiency or complicate the > compiler further I have no idea Actually isn't this a feature of cython? -- Armadillo: To provide weapons to a Spanish pickle. -- https://mail.python.org/mailman/listinfo/python-list
Re: Static typing [was Re: Python and the need for speed]
On Sun, 16 Apr 2017 14:27:28 +1000, Steve D'Aprano wrote: > On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote: > >> apparently, the py-devs believe we only deserve type declarations that >> do nothing to speed up code execution (aka: type-hints), instead of >> type declarations that could actually speed up the code. Go figure! >> >> I'm not a fan of forced static typing, but i am a fan of optional >> static typing. > > Are you aware that optional static typing CANNOT be used for > optimization? > > Since it is *optional*, it is only a hint, not a fact. You can tell the > compiler that you believe that n will be an int, but it's not > guaranteed. As the Dude says, the type hint is > > that's, like, only your opinion man > > and the compiler cannot trust it. It must still guard the operation with > a runtime check that n actually is an int, and so you lose most or all > of the benefit of telling the compiler that it's an int. > > There are clever Just In Time compiler tricks that can optimize the > code, which is what Psycho did and PyPy still does, but they don't need > type hints. Type hints are superfluous to a JIT compiler, since it > already knows the runtime type information. But JIT compilers have their > limitations, like the necessity of warm up time, increased complexity, > and much heavier memory requirements. > > If you want to use type hints for the purposes of compile-time > optimizations, then typing must be compulsory and enforced by the > compiler. > No longer optional hints, but mandatory statements of indisputable fact. > > (That doesn't necessarily mean that you have to use type declarations, > like in C, Pascal and Java. A good modern compiler can infer types, like > in ML and Haskell.) > > Python's gradual typing is not about optimization, it is about testing > program correctness and helping to find bugs at compile time*. As such, > the fact that type hints are optional and "only your opinion" doesn't > matter. The promise made by the type checker is: > > IF what you say is true (the type hints are correct) > THEN this code will be sound > > (or alternatively, it's unsound because you have a type mismatch, and > get a compile-time error). As a fallback, you still have the runtime > dynamic type checks. > > Unit and functional tests are great, and necessary. Just because your > code compiles in C, Java or Pascal doesn't mean that it is correct. You > still have to run the code and test that it does what you expect. > > "It compiles! Quick, ship it!" -- attributed to Microsoft developers > > Nevertheless, the problem with testing is that it can only prove the > presence of bugs, not their absence. A test that fails proves that your > code has at least one bug. A test that passes doesn't prove that there > are no bugs. > > Static type checking helps to shrink the gap between "all of my tests > pass, > so there are no known bugs" and "there are no unknown bugs". And that's > an important and powerful tool, especially for large and complex > programs. > > Python's type-hints are for writing correct code, not fast code. I find > it amusing when people argue that Python's type hints are pointless or > useless. Do they realise they're effectively arguing that correct, > bug-free code is pointless? > > "Your code is buggy. It calculates the wrong result." > > "Maybe so, but look how fast it does it!" I think you have (deliberately?) miss understood RR by optional Static typing I think he means you don't have to use it as is in suing static types is optional) but when you define a variable as static then its type is fixed, not simply a hint. whether or not this could increase code deficiency or complicate the compiler further I have no idea -- Fay: The British police force used to be run by men of integrity. Truscott: That is a mistake which has been rectified. -- Joe Orton, "Loot" -- https://mail.python.org/mailman/listinfo/python-list
Static typing [was Re: Python and the need for speed]
On Sat, 15 Apr 2017 11:55 am, Rick Johnson wrote: > apparently, the py-devs believe we > only deserve type declarations that do nothing to speed up > code execution (aka: type-hints), instead of type > declarations that could actually speed up the code. Go > figure! > > I'm not a fan of forced static typing, but i am a fan of > optional static typing. Are you aware that optional static typing CANNOT be used for optimization? Since it is *optional*, it is only a hint, not a fact. You can tell the compiler that you believe that n will be an int, but it's not guaranteed. As the Dude says, the type hint is that's, like, only your opinion man and the compiler cannot trust it. It must still guard the operation with a runtime check that n actually is an int, and so you lose most or all of the benefit of telling the compiler that it's an int. There are clever Just In Time compiler tricks that can optimize the code, which is what Psycho did and PyPy still does, but they don't need type hints. Type hints are superfluous to a JIT compiler, since it already knows the runtime type information. But JIT compilers have their limitations, like the necessity of warm up time, increased complexity, and much heavier memory requirements. If you want to use type hints for the purposes of compile-time optimizations, then typing must be compulsory and enforced by the compiler. No longer optional hints, but mandatory statements of indisputable fact. (That doesn't necessarily mean that you have to use type declarations, like in C, Pascal and Java. A good modern compiler can infer types, like in ML and Haskell.) Python's gradual typing is not about optimization, it is about testing program correctness and helping to find bugs at compile time*. As such, the fact that type hints are optional and "only your opinion" doesn't matter. The promise made by the type checker is: IF what you say is true (the type hints are correct) THEN this code will be sound (or alternatively, it's unsound because you have a type mismatch, and get a compile-time error). As a fallback, you still have the runtime dynamic type checks. Unit and functional tests are great, and necessary. Just because your code compiles in C, Java or Pascal doesn't mean that it is correct. You still have to run the code and test that it does what you expect. "It compiles! Quick, ship it!" -- attributed to Microsoft developers Nevertheless, the problem with testing is that it can only prove the presence of bugs, not their absence. A test that fails proves that your code has at least one bug. A test that passes doesn't prove that there are no bugs. Static type checking helps to shrink the gap between "all of my tests pass, so there are no known bugs" and "there are no unknown bugs". And that's an important and powerful tool, especially for large and complex programs. Python's type-hints are for writing correct code, not fast code. I find it amusing when people argue that Python's type hints are pointless or useless. Do they realise they're effectively arguing that correct, bug-free code is pointless? "Your code is buggy. It calculates the wrong result." "Maybe so, but look how fast it does it!" -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Looping [was Re: Python and the need for speed]
On Sat, 15 Apr 2017 10:17 pm, bartc wrote: > On 15/04/2017 03:35, Rick Johnson wrote: >> On Wednesday, April 12, 2017 at 8:44:30 AM UTC-5, bart...@gmail.com >> wrote: > >> At a minimum, every language should offer >> the following four loop-forms (using Python semantics): >> >> while CONDITION: >> doSomething() >> >> for VALUE in COLLECTION: >> doSomething(value) >> >> loop(N): >> doSomething() >> >> loop(N) as i: >>doSomething(i) What an extravagant, unnecessary waste of syntax. The last two just duplicate the functionality of the second. AND Rick has forgotten about the do...while form that executes the block at least once. I've used, and to be honest really enjoyed using, a language with multiple loop syntax like this. I can see some advantage in offering specialist English-like loops in a programming language for non-programmers and amateurs. But having a plethora of forms for what is essentially the same thing is not so helpful for more skilled programmers who are more comfortable with composing code rather than memorising fixed statements. repeat forever: ... is no more expressive than while True: ... since they're both just the same thing. You can mechanically and trivially translate one to the other with no intelligence needed. So it becomes a mere matter of personal taste as to which you prefer. > Yes, I'm constantly surprised at this, as such syntax has a very low > cost (in my last compiler, supporting 'while' for example only added 30 > lines to the project). That's the advantage of writing your own private language and having no users except for yourself. You get to cut corners. Python has tens of thousands of users, and doesn't have that luxury. For language developers with responsibilities to users, the job doesn't stop at just implementing the feature in the language. It also needs to be tested and documented. As a general rule of thumb, every line of production code should expect to add at least ten lines of test code (unit tests, regression tests, integration tests, functional tests, doc tests, etc). So for your "30 lines" feature, that adds 300 lines of tests, and probably another page or two of documentation: - describing the various syntax forms; - explaining how they differ; - tutorials for beginners showing each form; plus the on-going burden for every single user of the language, for ever, in having to decide which form they need to use in any specific circumstance. The more choices you offer, the harder that decision becomes: - numeric Pascal or C-style loop - foreach style loop - repeat while condition (test at start) - repeat until condition (test at start) - do ... while condition (test at end) - do ... until condition (test at end) - repeat forever I don't remember the language, but I remember seeing one generalisation of the repeat/do loop that puts the test in the middle, rather than at the start or end of the loop. If I remember it was something like: DO setup code # executed once only REPEAT loop body # before the test WHILE condition # test loop body # after the test thus combining both the repeat while condition: ... and do: ... until condition styles of looping in one handy syntax. > Of course, it's possible to overdo it; if you look at Lisp, you'll lose > yourself in the myriad looping options. "Myriad"? As I understand it, Lisp offers just *one* more loop construct than the number you agree is the "minimum" needed: five. loop loop for do dotimes dolist https://www.tutorialspoint.com/lisp/lisp_loops.htm > But very common requirements are endless loops, and repeat N times > without needing an explicit counter. If by "very common" you mean "occasionally", I agree. > The former /can/ be easily written > as: > > while 1: > body > > but it's more psychological; I don't want to use an idiom to denote an > endless loop, I want to be able to express it directly! "Express it directly" is an idiom, and "while True" is just as direct as "repeat forever". > Python's byte-code does at least optimise out the check that '1' is > true, but that's not what the reader sees, which is 'loop while 1 is > true'. And one day it will be: > > while l: > body > > that can be mistaken for that common idiom. The problem there is not the loop, but the foolish use of lowercase l as a variable name. It is simply poor programming practice to use easily confused names, and the problem here is not the use of `while`. No more than it is the use of `while` that makes this code bad: while O000IlI1III111IllOO: something() O000IlI1III11I1II1IllOO = False print(O00OIlI1III111IllOO) Given: mylist.append(l) x += l if l: ... do you also conclude that the problem lies with append, += and `if` ? -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure e
Re: Python and the need for speed
On Sun, 16 Apr 2017 12:37 am, bartc wrote: > What proportion of Python implementations depend on executing byte-code? My guess is 100%. Pretty much all modern interpreters of any language execute some form of byte-code or another. The bad old days where interpreters repeatedly parsed and executed each line of source are long gone. PyPy generates some sort of byte-code, with a JIT compiler to generate machine code on the fly for sufficiently "hot" code. IronPython uses byte-code for the .Net virtual machine. Jython uses byte-code for the Java virtual machine. CPython uses byte-code for its own virtual machine. There was a project for a Python interpreter for the Parrot VM, but I think that's no longer in development. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 04/15/2017 08:37 AM, bartc wrote: > What proportion of Python implementations depend on executing byte-code? Presumably Nuitka does not depend on any byte code at all. Jython uses JVM byte codes. Iron Python uses .net VM bytecodes. While CPython's byte codes do take their form in part because of Python and its syntax. But that doesn't mean that Python itself as a language is necessarily dependent on the specifics of the CPython byte code interpreter and architecture, though I concede that for some aspects of Python, the behavior of the language is defined by the CPython implementation. The two aspects are definitely linked, but not as solidly as you seem to think. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
bartc writes: > 'do', in the original Algol 68 syntax, was part of its 'for' > statement, where you could leave out the parts you don't need. The > full syntax is something like: > > for var := a to b by c while d do body od FOR name FROM e1 BY e2 TO e3 WHILE cond DO body OD The significant part is the the name is bound to a value and can't be assigned in the body. (You don't say it can be but "var := a" makes var look like a plain Algol68 variable.) -- Ben. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 15/04/2017 14:27, Marko Rauhamaa wrote: bartc : while 1: body Why not say it like it is: while True: body but it's more psychological; I don't want to use an idiom to denote an endless loop, I want to be able to express it directly! C's: for (;;) statement I would call an idiom. I don't like that either! Python's while True: body is about as straightforward as it gets. In the syntax I normally use (based on Algol 68), the endless loop is: do body end (well, actually I use do body od, but that may have raised some eyebrows). 'do', in the original Algol 68 syntax, was part of its 'for' statement, where you could leave out the parts you don't need. The full syntax is something like: for var := a to b by c while d do body od If you leave out the first part you get a while loop. Leave out all except 'to b do', you get a repeat-n-times loop. And so on. Elegant. (I don't do it that way, I have discrete loop types, plus my own variations. Devising syntax is so easy, the hard bit is knowing when to stop...) Python's byte-code does at least optimise I don't understand why there's so much talk about CPython's bytecode on this list. That should only be of interest to active CPython developers. What proportion of Python implementations depend on executing byte-code? -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Saturday, April 15, 2017 at 7:17:55 AM UTC-5, bartc wrote: > On 15/04/2017 03:35, Rick Johnson wrote: > > On Wednesday, April 12, 2017 at 8:44:30 AM UTC-5, bart...@gmail.com wrote: > > > At a minimum, every language should offer > > the following four loop-forms (using Python semantics): > > > > while CONDITION: > > doSomething() > > > > for VALUE in COLLECTION: > > doSomething(value) > > > > loop(N): > > doSomething() > > > > loop(N) as i: > >doSomething(i) > > > > Yes, I'm constantly surprised at this, as such syntax has a > very low cost (in my last compiler, supporting 'while' for > example only added 30 lines to the project). Of course, > it's possible to overdo it; if you look at Lisp, you'll > lose yourself in the myriad looping options. But very > common requirements are endless loops, and repeat N times > without needing an explicit counter. The former /can/ be > easily written as: > > while 1: > body > > but it's more psychological; I don't want to use an idiom > to denote an endless loop, I want to be able to express it > directly! A good point. I don't like the idiom of using "always True conditions" to denote an endless loop either. So it's just a matter of recognizing that loop() without an "explicit confining parameter", would be endless. loop: # This loop never ends. Of course, recognizing that at some point that the loop will need to end, or that something surprizing will cause it to end, how then would you end it in a clean manner? Which would cause the semantics to be: loop: # This loop never ends. Well "never" unless an # exception occurs; or the hardware malfunctions; or # melts; or unless nuclear war breaks out; or unless the # sun expands and envelopes the earth -- except for # those cases (and possibly a long list of others...), # it never ends. ;-) > Python's byte-code does at least optimise out the check > that '1' is true, but that's not what the reader sees, > which is 'loop while 1 is true'. And one day it will be: > > while l: > body > > that can be mistaken for that common idiom. Another good point. However that's more a consequence of using 1 as a truth value than anything else. Python has offered booleans for a lng time, so i don't understand why some of us feel the need to use 0 and 1 in place of the more intuitive (and less bug prone) False and True. The obvious solution is: while True: # This loop never ends. # (Note: see exceptions above) Integers should only be utilized for their numeric value, and nothing else. Python should never allow an integer to be used as a truth value, only booleans or expressions that evaluate to a truth value. Of course, these can perverted also. if 1 == 1: # superfluous! 1 will always equal 1 if None: # superfluous! None will always be False if True == False: # superfluous! True and False will never be equal. etc... (of course, i _hope_ those examples are rock solid. Sometimes, i forget how utterly and stubbornly dynamic this damned language can be!) -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
bartc : > Of course, it's possible to overdo it; if you look at Lisp, you'll lose > yourself in the myriad looping options. Funny, in Scheme, the only looping construct I use is named-let. > The former /can/ be easily written as: > > while 1: > body Why not say it like it is: while True: body > but it's more psychological; I don't want to use an idiom to denote an > endless loop, I want to be able to express it directly! C's: for (;;) statement I would call an idiom. Python's while True: body is about as straightforward as it gets. > Python's byte-code does at least optimise I don't understand why there's so much talk about CPython's bytecode on this list. That should only be of interest to active CPython developers. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Sat, 15 Apr 2017 10:23 am, bartc wrote: > On 15/04/2017 00:40, Rick Johnson wrote: >> * `range(10)` will always produce a list of the _same_ 10 >> integers. > > You don't know if 'range' is still a range. If this has been executed > first, then the answer will be 20: > > oldrange=range > def double(x): return oldrange(x*2) > range=double Correct. It is true that *at the moment* CPython doesn't know if range has been shadowed or redefined, but Victor Stinner is working on a project, FATPython, which will. Or at least, FATPython can tell: - there's no chance that range() has been redefined, so it is safe to optimize the call to it; - or there is a chance, in which case it is not safe to optimize. >> * `for i in range(10)` will aways iterate over the _same_ >> 10 integers > > And range could be set to something else each time the loop is executed. It could, but that won't effect the loop. py> old_range = range py> for i in range(5): ... range = i*100 ... print(i) ... 0 1 2 3 4 py> range 400 >> * `answer += 1` will always increment the "current integer >> value" by 1 > > I don't know if it's possible to override __iadd__ for integers so that > +=1 does something different here. No, Python doesn't allow the built-in types to be monkey-patched. However, that's not the problem. The problem is that CPython doesn't know whether answer is a built-in int or not until runtime. -- Steve “Cheer up,” they said, “things could be worse.” So I cheered up, and sure enough, things got worse. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On 15/04/2017 03:35, Rick Johnson wrote: On Wednesday, April 12, 2017 at 8:44:30 AM UTC-5, bart...@gmail.com wrote: At a minimum, every language should offer the following four loop-forms (using Python semantics): while CONDITION: doSomething() for VALUE in COLLECTION: doSomething(value) loop(N): doSomething() loop(N) as i: doSomething(i) Yes, I'm constantly surprised at this, as such syntax has a very low cost (in my last compiler, supporting 'while' for example only added 30 lines to the project). Of course, it's possible to overdo it; if you look at Lisp, you'll lose yourself in the myriad looping options. But very common requirements are endless loops, and repeat N times without needing an explicit counter. The former /can/ be easily written as: while 1: body but it's more psychological; I don't want to use an idiom to denote an endless loop, I want to be able to express it directly! Python's byte-code does at least optimise out the check that '1' is true, but that's not what the reader sees, which is 'loop while 1 is true'. And one day it will be: while l: body that can be mistaken for that common idiom. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Thursday, April 13, 2017 at 1:32:28 AM UTC-5, Steven D'Aprano wrote: > On Wed, 12 Apr 2017 14:38:52 +0100, Ben Bacarisse wrote: > > Steve D'Aprano writes: > >> On Wed, 12 Apr 2017 03:39 am, Paul Rubin wrote: > > [...] Indeed, and this is a very common phenomenon: > features which "ordinary" programmers imagine are "trendy" > and new end up having their roots in some functional > language dating back to the 1980s, or 70s, or in extreme > cases the 1950s and Lisp. Be advised that functional programmers are a fanatical bunch, and saying something like that could be akin to blowing a "functional purist dogwhistle". If this thread becomes overrun with functional trolls, i'm blaming you. ;-) -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Thursday, April 13, 2017 at 1:32:29 AM UTC-5, Gregory Ewing wrote: > Jussi Piitulainen wrote: > > Traceback (most recent call last): > > File "/dev/fd/63", line 37, in > > SanityClauseException: code is blatantly sub-optimal > > > > As far as I know, no language does that. Because reasons? > > Because the problem of making a compiler do that is > probably AI-complete! What? GvR's time machine only goes backwards? -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Wednesday, April 12, 2017 at 8:44:30 AM UTC-5, bart...@gmail.com wrote: > On Wednesday, 12 April 2017 12:56:32 UTC+1, Jussi Piitulainen wrote: > > bartc writes: > > > > > > These are straightforward language enhancements. > > > > FYI, the question is not how to optimize the code but how > > to prevent the programmer from writing stupid code in the > > first place. Someone suggested that a language should do > > that. > The 'stupid code' thing is a red herring. I assume the > code people write is there for a reason. Yeah, but you have to admit, sometimes there is no obvious good reason -- as Steven's example code has proven. ;-) Although, i'm convinved that any one of us could find examples like that in our own repos. And if you cannot find code that makes you feel embarassed, then you're not evolving anymore. > But the language can also play a part in not allowing > certain things to be expressed naturally. Exactly! > So the for-loop in the example has to have a control- > variable even if it's not referenced. Yes, one of the design flaws of Python's "for loop" is that the control variable is injected whether you need it or not (Nightmarish images of petulant children being force-fed castor oil comes to mind...). In fact, in Python, there are only two loop forms avialable -- the "for" and the "while". -- however, many other languages have relized that these two forms are insufficent to cover the various types of loops that are needed. At a minimum, every language should offer the following four loop-forms (using Python semantics): while CONDITION: doSomething() for VALUE in COLLECTION: doSomething(value) loop(N): doSomething() loop(N) as i: doSomething(i) -- https://mail.python.org/mailman/listinfo/python-list
Re: Python and the need for speed
On Wednesday, April 12, 2017 at 4:57:10 AM UTC-5, bart...@gmail.com wrote: > On Wednesday, 12 April 2017 07:48:57 UTC+1, Steven D'Aprano wrote: > > On Tue, 11 Apr 2017 21:10:56 -0700, Rick Johnson wrote: > > > > > > high level languages like Python should make it > > > difficult, if not impossible, to write sub- optimal code > > > (at least in the blatantly obvious cases). > > > > Here's another example: > > > > answer = 0 > > for i in range(10): > > answer += 1 > > > > instead of > > > > answer = 10 > > > > So... how exactly does the compiler prohibit stupid code? > > Actually, an optimising C compiler (not one of mine!) > probably could reduce that to answer=10. And eliminate even > that if 'answer' was never used. > > But that's not really the issue here. Assume that such a > loop /is/ doing something more useful. The problems with > Python (last time I looked anyway) were these: > > (1) If answer was a global variable, then it needs a symbol > table lookup to find out what it is. That would not happen > in a static language, or a less dynamic one, as you already > have the address. Indeed. I have argued for syntax that will resolve variable scope many times. but apparently, the py-devs believe we only deserve type declarations that do _nothing_ to speed up code execution (aka: type-hints), instead of type declarations that could actually speed up the code. Go figure! I'm not a fan of forced static typing, but i am a fan of optional static typing. The global keyword is one idiosyncrasy of Python that causes a lot of confusion , especially to noobs, but also from a standpoint of "general intuitiviness" for old hats. This keyword does not define "true globals" (aka: a symbol that can be accessed or reassigned from anywhere), no, but rather, a symbol that is global only with respect to the "current module scope" (aka: module level variable). You can only get "true globals" in Python by going rogue and injecting symbols into sys.modules like some mad scientist -- which is not officially supported. Go figure! ;-) > And this [global] lookup happens for every loop iteration. I sure hope not. I have not taken the time to inspect the inner workings of Python, but if the lookup happens every time, that would be an awfully inefficient design. > (2) There is also 'range', which could have been redefined > to mean something else, so /that/ needs a lookup. The byte- > code compiler can't just assume this loop is to be executed > 10 times. Yep, and more evidence that Python has taken dynamics to such a fundamentalist extreme, that even ISIS is jealous! # START INTERACTIVE SESSION (Py2.x) ## >>> def range(*args): ... return "Death to static infidels!" >>> for i in range(10): ... print i D e a t h t o s t a t i c i n f i d e l s ! ## END INTERACTIVE SESSION ## > (3) This was fixed long ago but at one time, even when > 'range' had been established to be a range, it involved > constructing a list of items (10 here, but it could be a > million), and then iterating over the list. First, it was "range", then "xrange", and now, "range" again. Third time's a charm i suppose. The last iteration of the range function design removed the underlying inefficiency, however, it created a semantical nightmare (more on this below). > This might seem crazy, but it might have been exceptable > for a script language at one time. Not for a general > purpose one however. Yeah, this is one of those screw-ups that even GvR's famous time machine could not fix. Must have been in the shop for a tune-up... > (4) Python's integer types being immutable, the += > operation means evaluating the result, then creating a new > integer object and binding 'a' to that new value. (Correct > me if I'm wrong.) > > These are all things the language could point a finger at > before blaming the user for writing inefficient code. > > The problem is also the language encouraging people to use > high-level but inefficient methods, as the emphasis is on > productivity and readability** rather than performance. In > fact most users probably have no idea on what is efficient > and what isn't. And that's the point i was trying to make earlier. Most Python programmers are not CS majors, they are just "technical folk" who need to write a logic program to do this or that task. We shouldn't expect them to know about things like memory management, neither should we expect/burden them to know that calling range(100), just to do one million iterations, is inefficient. Instead, we should provide them with the proper "efficient tools" to get the job done. So instead of: for x in range(100): # Legacy python was slow here! doSomething(x) It should have _always_ been... 100.times: doSomething() or, if you need a counter: 100.times as x: doSomething(x) Or som
