Status of side-effecting functions in python
Moved from other (Seymore's) thread where this is perhaps not relevant On Saturday, October 25, 2014 1:15:09 PM UTC+5:30, Steven D'Aprano wrote: > Rustom Mody wrote: > > > On Saturday, October 25, 2014 11:20:03 AM UTC+5:30, Chris Angelico wrote: > >> On Sat, Oct 25, 2014 at 4:40 PM, Rustom Mody wrote: > >> > Its generally accepted that side-effecting functions are not a good > >> > idea -- typically a function that returns something and changes global > >> > state. > >> > >> Only in certain circles. Not in Python. There are large numbers of > >> functions with side effects (mutator methods like list.append, > >> anything that needs lots of state like random.random, everything with > >> external effect like I/O, heaps of stuff), and it is most definitely > >> not frowned upon. > >> > >> In Python 3 (or Python 2 with the future directive), print is a > >> function, print() an expression. It's not "semantically a statement". > > > > Ok > > So give me a valid (ie useful) use where instead of the usual > > l=[1,2,3] > > l.append(4) > > > > we have > > > > foo(l.append(4)) > > Your question seems to be non-sequitor. To me, it doesn't appear to have any > relationship to Chris' comments. | Languages like Pascal (many others)... distinguish function which return | results and procedure which do not... | Extract from https://en.wikipedia.org/wiki/Subroutine#Language_support So my point: Whether the language supports it strongly (Pascal' procedures) weakly (C's void functions) more weakly (Python's None returning functions), the beginning programmer needs this concept as a core thinking tool. Pascal makes this easy -- teach the syntax and the concept will get across Python is harder because the concept does not correlate with any specific syntax But its not all that hard unless the teacher is stuck in correlating core concepts and language syntax. A teacher who is so stuck is cheating the student. My version: "print may (3) or may not (2) be an expression. Just always consider it as a statement" Chris version: print() is an expression. Technically Chris is correct. Is it methodologically/pedagogically it is sound? Consider: >From my explanation this: >>> [print(x) for x in [1,2,3]] 1 2 3 [None, None, None] >>> is in "Dont Do!" category whether python (3) allows it or python 2 doesn't. And you "Dont do" because print(x) is a statement -- literally in python 2; morally in python 3 And from here its only a small step to why l.append(4) should never be used except as a statement. Python may not literally have Pascal's procedures; they are morally there if you choose to 'believe' in them. How would Chris inculcate avoidance of code-smells like - foo(l.append(4)) - [print(x) for x in [1,2,3]] ? No idea! I dare say: about print I dont know but [foo(x) for x in lst] and foo is used for effect not return-value is a rather common code-smell in my experience -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
On 25.10.2014 19:27, Rustom Mody wrote:
Moved from other (Seymore's) thread where this is perhaps not relevant
On Saturday, October 25, 2014 1:15:09 PM UTC+5:30, Steven D'Aprano wrote:
Rustom Mody wrote:
On Saturday, October 25, 2014 11:20:03 AM UTC+5:30, Chris Angelico wrote:
On Sat, Oct 25, 2014 at 4:40 PM, Rustom Mody wrote:
Its generally accepted that side-effecting functions are not a good
idea -- typically a function that returns something and changes global
state.
Only in certain circles. Not in Python. There are large numbers of
functions with side effects (mutator methods like list.append,
anything that needs lots of state like random.random, everything with
external effect like I/O, heaps of stuff), and it is most definitely
not frowned upon.
In Python 3 (or Python 2 with the future directive), print is a
function, print() an expression. It's not "semantically a statement".
Ok
So give me a valid (ie useful) use where instead of the usual
l=[1,2,3]
l.append(4)
we have
foo(l.append(4))
Your question seems to be non-sequitor. To me, it doesn't appear to have any
relationship to Chris' comments.
| Languages like Pascal (many others)... distinguish function which return
| results and procedure which do not...
| Extract from https://en.wikipedia.org/wiki/Subroutine#Language_support
So my point: Whether the language supports it strongly (Pascal'
procedures) weakly (C's void functions) more weakly (Python's None
returning functions), the beginning programmer needs this concept as a core
thinking tool.
Pascal makes this easy -- teach the syntax and the concept will get across
Python is harder because the concept does not correlate with any specific syntax
But its not all that hard unless the teacher is stuck in correlating core
concepts
and language syntax.
A teacher who is so stuck is cheating the student.
My version:
"print may (3) or may not (2) be an expression. Just always consider it as a
statement"
Chris version: print() is an expression.
Technically Chris is correct. Is it methodologically/pedagogically it is sound?
Consider:
From my explanation this:
[print(x) for x in [1,2,3]]
1
2
3
[None, None, None]
is in "Dont Do!" category whether python (3) allows it or python 2 doesn't.
And you "Dont do" because print(x) is a statement -- literally in
python 2; morally in python 3
And from here its only a small step to why l.append(4) should never
be used except as a statement. Python may not literally have Pascal's
procedures;
they are morally there if you choose to 'believe' in them.
How would Chris inculcate avoidance of code-smells like
- foo(l.append(4))
- [print(x) for x in [1,2,3]]
As Chris and Steven have pointed out, picking print() as an example does
not make too much sense since it returns None.
It may be rare to use an expression both for its side-effects and its
return value, but, provided you document the intention appropriately, I
do not see what would generally be wrong with it.
A quick example that's not quite as silly as all your print() ones:
>>> with open('longnumber.txt', 'w') as out:
print(sum(out.write(str(x)) for x in range(100)), 'characters written.')
190 characters written.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
On Sat, 25 Oct 2014 23:41:52 +0200, Wolfgang Maier wrote: > ... It may be rare to use an expression both for its side-effects and > its return value ... A lot of concurrency-related operations work that way. In the old days, it was CPU-level Test and Set (or Compare and Set) instructions. These days, we have Python's threading.Lock.acquire. Dan -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
On 10/25/2014 6:22 PM, Dan Sommers wrote: On Sat, 25 Oct 2014 23:41:52 +0200, Wolfgang Maier wrote: ... It may be rare to use an expression both for its side-effects and its return value ... A lot of concurrency-related operations work that way. In the old days, it was CPU-level Test and Set (or Compare and Set) instructions. These days, we have Python's threading.Lock.acquire. list.pop and and related set and dict methods. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
On Sun, 26 Oct 2014 00:45:49 -0700, wxjmfauth wrote:
> Ditto for .write(). Why should it return "something" ?
>
with open('z.txt', 'w') as f:
> ... f.write('abc')
> ...
> 3
OTOH, why shouldn't it return something? In this case, it returns the
length of the string written. This value is analogous to the value
returned by the underlying OS function (at least on a POSIX-like system,
where write(2) returns the number of bytes written). This value can be
useful for detecting when things have gone wrong; e.g., disk full,
network down, pipe broken, etc. Practicality definitely beats purity.
At one time, on a huge project, millions of lines of C and assembly
code, we had a local guideline *not* to write void functions. The idea
was to return something that might be useful later, even if it seemed
unlikely now. A simple success flag was sufficient; as functions grew,
often did their failure modes.
Dan
--
https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
On Sunday, October 26, 2014 7:11:43 PM UTC+5:30, Dan Sommers wrote: > At one time, on a huge project, millions of lines of C and assembly > code, we had a local guideline *not* to write void functions. The idea > was to return something that might be useful later, even if it seemed > unlikely now. A simple success flag was sufficient; as functions grew, > often did their failure modes. Well C and Python are completely different in this respect. In C it is the norm to return the status in the return value and the actual return value in pointer parameters. Or even worse in pointer parameters + globals: think of most of the system calls and errno. This low-level messiness is in fact one of the compulsions that have driven preference and development of higher level languages like python. Here failure-mode has dedicated syntax -- exceptions. This conduces to a more DRY, less error-prone setup. - Normal mode in the normal functional call-return style - Failure mode in the except (for caller); raise (for callee) clauses. Note that my comment was: > Its generally accepted that side-effecting functions are not a good idea > -- typically a function that returns something and changes global state. So I am not talking of merely global (or non-local) variable side-effecting functions/methods which is normal in imperative programming, but ones that do BOTH - both changing global state - returning useful results in the return value While this may be called a code-smell, in C like languages it is close to unavoidable. In python its more avoidable and therefore more smelly (to my nose at least!) Yeah I note Terry's examples of list.pop etc. I guess one could say that these are instances of practicality beats purity. However note the context where this thread arose. A beginning programmer having a hard time distinguishing: - Values and effects - Expressions and statements - Immutable and mutable data-structures - Functions and Procedures My claim is that until then, he should not be trying to write code like that. -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
wxjmfa...@gmail.com:
> Yes and no. If something goes wrong in a .write() method,
> is not Python supposed to raise an error? (!)
We have multiple cases:
1. write succeeds with all of the given bytes
2. write succeeds with some but not all of the given bytes
3. write cannot at the moment write any bytes
4. an I/O error has taken place
Cases 1 and 2 are reported through positive return values in POSIX
system calls. Cases 3 and 4 are reported as errors. Arguably 2 and 3 are
related cases.
Python imitates POSIX, but *could* take a different tack. For example,
* always return with an exception carrying the number of bytes written
* return with None in case 1 and with an exception otherwise
* return with the number of bytes in 1, 2 and 3 and exception in 4
The POSIX way is justified with symmetry: read() distinguishes EAGAIN
from 0 (end of file). One could argue that POSIX got it the wrong way.
EOF should be an exception ("EEOF") and 0 should be used instead of
EAGAIN. However, the POSIX practice works in C and Python.
Marko
--
https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
In article <683c84d8-d916-4b63-b4b2-92cd2763e...@googlegroups.com>,
wxjmfa...@gmail.com wrote:
> Le dimanche 26 octobre 2014 14:41:43 UTC+1, Dan Sommers a écrit :
> > On Sun, 26 Oct 2014 00:45:49 -0700, wxjmfauth wrote:
> >
> > > Ditto for .write(). Why should it return "something" ?
> > >
> > with open('z.txt', 'w') as f:
> > > ... f.write('abc')
> > > ...
> > > 3
> >
> > OTOH, why shouldn't it return something? In this case, it returns the
> > length of the string written. This value is analogous to the value
> > returned by the underlying OS function (at least on a POSIX-like system,
> > where write(2) returns the number of bytes written). This value can be
> > useful for detecting when things have gone wrong; e.g., disk full,
> > network down, pipe broken, etc. Practicality definitely beats purity.
> >
> > At one time, on a huge project, millions of lines of C and assembly
> > code, we had a local guideline *not* to write void functions. The idea
> > was to return something that might be useful later, even if it seemed
> > unlikely now. A simple success flag was sufficient; as functions grew,
> > often did their failure modes.
> >
> > Dan
>
> Yes and no. If something goes wrong in a .write() method,
> is not Python supposed to raise an error? (!)
Define "wrong". It is not an error for a write() call to consume fewer
bytes than were requested. How would you expect this to be handled in
Python? Raise DataPartiallyWrittenError?
--
https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
Roy Smith wrote: >> Yes and no. If something goes wrong in a .write() method, >> is not Python supposed to raise an error? (!) > > Define "wrong". It is not an error for a write() call to consume fewer > bytes than were requested. It's not? I'm asking a genuine question here, not a rhetorical one. I would expect that if I ask to write 2 bytes, and only 1 byte is written, that absolutely is an error. Under what circumstances is it okay for write() to throw data away? > How would you expect this to be handled in Python? Raise > DataPartiallyWrittenError? I would expect it to raise an IOError, most likely with one of the following error codes: * errno.EIO (physical input/output error) * errno.EFBIG (file is too large) * errno.ENOSPC (no space left on device, disk is full) -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
On Mon, Oct 27, 2014 at 10:30 PM, Steven D'Aprano wrote: > Roy Smith wrote: > >>> Yes and no. If something goes wrong in a .write() method, >>> is not Python supposed to raise an error? (!) >> >> Define "wrong". It is not an error for a write() call to consume fewer >> bytes than were requested. > > It's not? I'm asking a genuine question here, not a rhetorical one. I would > expect that if I ask to write 2 bytes, and only 1 byte is written, that > absolutely is an error. Under what circumstances is it okay for write() to > throw data away? > >> How would you expect this to be handled in Python? Raise >> DataPartiallyWrittenError? > > I would expect it to raise an IOError, most likely with one of the following > error codes: > > * errno.EIO (physical input/output error) > > * errno.EFBIG (file is too large) > > * errno.ENOSPC (no space left on device, disk is full) You're assuming the condition, whatever it is, is permanent. The most common reason for write() to be temporarily unable to write everything is a non-blocking socket, pipe, or somesuch. It writes as much as it can, tells you how much that is, and lets you buffer the rest or deal with it in whatever other way you choose. If it is permanent, though, then yes, it should tell you. But what if you ask it to write a megabyte, it writes half of it, and then finds that there's no space on the disk? Should it: 1) Back out the write and raise an exception? 2) Write part of the data and raise an exception? 3) Write part of the data and NOT raise an exception? All three make sense. The third one is an option only if it's documented as being able to tell you about partial writes. The second has a problem in that you can't necessarily communicate "this is how much I wrote" properly while also signalling the exception (imagine if one function calls write() more than once, and it doesn't catch any exceptions, just lets them bubble up). And backing out a write isn't always possible. So what's to do? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
In article <544e2cf2$0$13009$c3e8da3$54964...@news.astraweb.com>, Steven D'Aprano wrote: > Roy Smith wrote: > > >> Yes and no. If something goes wrong in a .write() method, > >> is not Python supposed to raise an error? (!) > > > > Define "wrong". It is not an error for a write() call to consume fewer > > bytes than were requested. > > It's not? I'm asking a genuine question here, not a rhetorical one. I would > expect that if I ask to write 2 bytes, and only 1 byte is written, that > absolutely is an error. Under what circumstances is it okay for write() to > throw data away? It's not throwing away data. The write() call returns a count of how many bytes is consumed, so you can present the rest of them again in a later write() call (assuming that makes sense to do for your application). In some cases, the underlying hardware (or network protocol) may be unable to handle the number of bytes you requested, or may fail in mid-transmission. Imagine a serial link. You tell it to write 100 bytes. It starts sending them down the line and after 20 bytes, the connection fails. What should write() do in that case? It hasn't written all the data, so it needs to let you know that. It also has written *some* of the data, so it needs to let you know that too. What you do with that information is up to you, but it clearly needs to return a richer status indication than just success/failure. -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
On 2014-10-25, Wolfgang Maier wrote:
> It may be rare to use an expression both for its side-effects and its
> return value,
It's actually quite common.
For example:
f = open("filename")
d = f.readline()
In both of those lines, the side effects and return values are equally
vital. The same applies to reading from a queue, popping from a
stack, etc.
--
Grant Edwards grant.b.edwardsYow! This PORCUPINE knows
at his ZIPCODE ... And he has
gmail.com"VISA"!!
--
https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
On 2014-10-27, Steven D'Aprano wrote: > Roy Smith wrote: > >>> Yes and no. If something goes wrong in a .write() method, >>> is not Python supposed to raise an error? (!) >> >> Define "wrong". It is not an error for a write() call to consume fewer >> bytes than were requested. > > It's not? I'm asking a genuine question here, not a rhetorical one. No. Under Unix/Posix a write() call may _always_ write fewer bytes than requested. It may be that the device/pipe/file whatever has filled and blocking is disabled. It may be that the device has decided that, at the moment, it can only handle a certain amount of data for some other reason. For example: Let's say you're writing to a network connection that must segment data, and you try to write more than will fit in the current segment. The write() call may fill the segment, send the segment, and "refuse" the rest of the data -- requiring that you make a subsequent write() with the remaining data (at which point it will start a new segment). Or, it may be because the system call was interrupted by something completely unrelated to your program, the write() call, or the "thing" to which you're writing [and it was more convenient for whoever wrote the OS to do a partial write than it was to try to resume the write]. If you really want to make sure that all bytes get written, you _must_ put all write() calls in a loop that checks the return value and keeps re-writing any unwritten data. And to answer your next question: yes, Unix application programmers have been complaining about that (perhaps justifiably) since 1970. -- Grant Edwards grant.b.edwardsYow! I have a TINY BOWL in at my HEAD gmail.com -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
Grant Edwards : > If you really want to make sure that all bytes get written, you _must_ > put all write() calls in a loop that checks the return value and keeps > re-writing any unwritten data. > > And to answer your next question: yes, Unix application programmers > have been complaining about that (perhaps justifiably) since 1970. I wouldn't have it any other way. Now, I have confused the discussion with some misinformation myself. Python2's file.write() doesn't return a value but pushes the whole string out. Python3's file.write() returns the number of *characters* written. I don't know if the number can ever be different from the total number of characters in the string. In POSIX, a write(2) system call on file blocks until all bytes have been passed on to the file system. The only exception (no pun intended) I know is the reception of a signal. Even then, I'm not sure Linux file systems ever cut writes short because of signals. I think the lack of nonblocking file access in Linux is one of the OS's main shortcomings. Python's sockets and pipes don't have write methods. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
On Mon, 27 Oct 2014 17:14:58 +0200, Marko Rauhamaa wrote: > In POSIX, a write(2) system call on file blocks until all bytes have been > passed on to the file system. The only exception (no pun intended) I know > is the reception of a signal. Writing to a file (or block device) will return a short count in the event that it results in the size of the file exceeding * the space available on the partition, * the user's quota, * the process' file size limit (RLIMIT_FSIZE), or * any implementation limit on the maximum size of a file, and at least one byte can be written. This behaviour is mandated by POSIX. This is different to writing to a socket, pipe or character device, where a short count is considered an entirely normal result, and a subsequent write for the remaining bytes will often succeed. > Even then, I'm not sure Linux file systems ever cut writes short because > of signals. Linux never interrupts I/O on discs or block devices due to signals. These are restarted regardless of whether the signal is set for automatic restarting (SA_RESTART flag). > I think the lack of nonblocking file access in Linux is one of the OS's > main shortcomings. It doesn't really matter. In the absence of an explicit mlock() or mlockall(), the actual code which would be controlling the access is demand-paged from disc, as is the "memory" to/from which the data is transferred (along with the memory which would hold the return code from read() or write(), for that matter). Asynchronous I/O in the sense of select(), poll(), O_NONBLOCK etc is meant for situations where delays could be indefinite, e.g. network connections or terminals. For "short" delays (i.e. disc access), there's not much point having a mechanism so that you can avoid blocking while the data is read from disc just so that you can block while the code in the "else" branch is read from disc. If you want the program to be able to do something else while waiting for I/O, use threads. The introduction of threads made most concurrency- related issues in the POSIX API moot. -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
Nobody : > Asynchronous I/O in the sense of select(), poll(), O_NONBLOCK etc is > meant for situations where delays could be indefinite, e.g. network > connections or terminals. For "short" delays (i.e. disc access), > there's not much point having a mechanism so that you can avoid > blocking while the data is read from disc just so that you can block > while the code in the "else" branch is read from disc. I disagree with the "shortness" of the delays. > If you want the program to be able to do something else while waiting > for I/O, use threads. The introduction of threads made most > concurrency- related issues in the POSIX API moot. I disagree about that point of view as well. If files played ball with select() et al, the linux world would be a better, more coherent place. For example, looking at Python3's asyncio, I don't want to go out of the asyncio model just because of disk (or DB) access. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Status of side-effecting functions in python
Marko Rauhamaa : > Python's sockets and pipes don't have write methods. Actually, that's mistaken as well. The sys.std* handles and pipes returned by subprocess are accessed using file.write() and thus may return partial writes. That brings up another point: Python3's file.write() returns the number of characters written. What might it return if a partial character should be written? Marko -- https://mail.python.org/mailman/listinfo/python-list
