[Python-ideas] Re: Python array multiplication for negative values should throw an error
On Mon, May 30, 2022 at 02:31:35PM -, fjwillemsen--- via Python-ideas wrote: > In Python, array multiplication is quite useful to repeat an existing > array, e.g. [1,2,3] * 2 becomes [1,2,3,4,5,6]. It certainly does not do that. >>> [1, 2, 3]*2 [1, 2, 3, 1, 2, 3] Also, that's a list, not an array. Python does have arrays, from the `array` module. And of course there are numpy arrays, which behave completely differently, performing scalar multiplication rather than sequence replication: >>> from numpy import array >>> array([1, 2, 3])*2 array([2, 4, 6]) > However, operations such as [numpy array] * -1 are very common to get > the inverse of an array of numbers. If that is common, there's a lot of buggy code out there! *wink* Multiplying a numpy array by the scalar -1 performs scalar multiplication, same as any other scalar. To get the inverse of a numpy array, you need to use numpy.linalg.inv: >>> import numpy.linalg >>> arr = array([[1, 2], [3, 4]]) >>> numpy.linalg.inv(arr) array([[-2. , 1. ], [ 1.5, -0.5]]) > The confusion here stems from the lack of type checking: while the > programmer should check whether the array is a NumPy array or a Python > array, this is not always done, giving rise to difficult to trace > cases where [1,2,3] * -1 yields [] instead of [-1,-2,-3]. This confusion has nothing to do with multiplication by -1. As the earlier example above shows, scalar multiplication on a numpy array and sequence replication on a list always give different results, not just for -1. (The only exception is multiplication by 1.) I am afraid that this invalidates your argument from Numpy arrays. It simply isn't credible that people are accidentally passing lists instead of numpy arrays, and then getting surprised by the result **only** when multiplying by a negative value. Its not just negatives that are different. > I can not think of good reasons why Python array multiplication should > not throw an error for negative multipliers, because it is meaningless > to have array multiplication by negative value in the way it is > intended in Python. Its not meaningless, it is far more *useful* than an unnecessary and annoying exception would be. For example, here is how I might pad a list to some minimum length with zeroes: mylist.extend([0]*(minlength - len(mylist))) If this was 1991 and Python was brand new, then the behaviour of sequence replication for negative values would be up for debate. But Python is 31 years old and there is 31 years worth of code that relies on this behaviour, so we would need **extraordinarily strong** reasons to break all that code. Not an extraordinarily weak argument based on confusion between numpy array scalar multiplication and list replication. Sorry to be blunt. -- Steve ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4OVPWC35WCGLNKSACH5PWWM77XR7425P/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Opt-in “iter def” and/or “gen def” syntax for generator functions
I don't really disagree with most of what you wrote! And agree that decorators, specifically, are a pretty good solution within the scope of an individual package. But I would quibble with this: >How fundamental is it that THIS function is a generator, rather than simply >that > it returns an iterator (or that it returns a generator/coroutine object, etc)? The delayed execution of a generator function is, IMO very, very different than a regular function! ``` def f(): result = side_effect1() yield 0 yield 1 return result ``` vs. ``` def f(): side_effect() return range(2) ``` These are contrived examples, obviously, but I've specifically encountered issues like this when working with code that handles streaming data - from a db or REST API, for instance. My experience of this type of code is often along the following lines: ``` def stream_rows(self): [mutex handling ...] [retry logic] [more mutexes?] [some timeout stuff] [...] ``` I don't mean to belabor the point - the decision in PEP 255 is pretty definitive, and this was more or less a "modest proposal" anyway. I do appreciate the responses and discussion! Thanks, Aaron ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/JMAWFCPYBW5BN2C33KNBE563YHRZZPXG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Python array multiplication for negative values should throw an error
On Wed, 1 Jun 2022 at 03:41, Christopher Barker wrote: > In fact, I don’t think there’s been a breaking change since the Py3 > transition. > > I think PEP 563 was going to be the first — and that’s been delayed. Depends on your definition of "breaking". For instance, this code is legal in Python 3.9, gives a DeprecationWarning in 3.10, and will eventually be illegal: >>> 5in range(10) True Technically, that's a breaking change. It's currently legal, and eventually won't. But (a) very little code will use this intentionally, and (b) it's trivially easy to fix it in a way that works on older versions (since the more normal way of writing it, with a space before the word "in", is compatible with every version of Python known to mankind, or thereabouts). Similarly, collections.Mapping has been deprecated since 3.3 (use collections.abc.Mapping instead), and that's no longer legal. It, too, gave a warning for a couple of versions. Generally, with every new Python version, it's a good idea to check the What's New for anything that's been removed, but for the most part, code won't break, or if it does, it'll be only in very minor ways. (One thing that Python really tries extremely hard to avoid is behavioural changes, where the same syntax is legal in multiple versions, but does different things. Those are extremely difficult to cope with in cross-version code.) Making something illegal that was previously valid is definitely possible, but it needs strong justification. Given that the "was this a list or a numpy array?" question can never be answered safely when you multiply by a positive integer, I'm dubious of the value of detecting an error when you multiply by a negative integer. It would be a backward compatibility break for fairly low value, in my opinion. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HQBOEZKR3YVRIXII3XGGYQW2QN43EQLT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Opt-in “iter def” and/or “gen def” syntax for generator functions
On Wed, 1 Jun 2022 at 03:34, Aaron L via Python-ideas wrote: > > Thanks for your reply. > > > What's the advantage? > > I brought this up thinking about explicitness and readability. Say you want > to figure out what this function is doing: > > > def foo() -> t.Iterator[T]: > [... 300 lines of code] > ``` > > Is this a generator function? I'd argue that whether it's a generator > function or not is fundamental to being able to read it. The type hint alone > doesn't tell you whether you're looking at a generator function or not - it > might just construct and return an iterator. > Does it actually matter whether it's a generator, or returns some other type of iterable? What's the difference between these two functions: def enumerate_spam(n): yield "spam 1" yield "spam 2" yield "spam 3" yield n yield "the rest of the spam" def enumerate_default_spam(): return enumerate_spam("default") Technically, one of these is a generator, and one is not. But the return value from both of them is a generator object. You can send it values, get values back, all the things you can do with a generator. How fundamental is it that THIS function is a generator, rather than simply that it returns an iterator (or that it returns a generator/coroutine object, etc)? If your function is really just named "foo" and has 300 lines of code, you have other problems. Normally, the function's name should tell you a lot. In your case, you seem to also have a return type hint, which tells you a bit more, so that ought to be sufficient? Maybe that's not sufficient for your codebase. Well, that's what docstrings and decorators and code comments are for :) ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7DC4LRAJXIIAPH43NCFJJIAKUG4M6RPD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Python array multiplication for negative values should throw an error
Two contradictory opinions from me :-) 1) multiplying a list or a bumpy array be an integer are very different operations — and in most cases, the result will be a different size (and type), which is the case in this example. So it’s rarely a hard to find bug. So that’s not a very compelling argument for this change. 2) this is surprising result that I’d pretty surprised that much code is using on purpose. So I’d think that it would make sense to have it raise an exception. Unless of course, someone does come up with some evidence that there is code in the wild that is counting on it working this way. BTW: to the OP: yes, Python can change, but we do try to keep breaking changes to an absolute minimum, and any breaking change needs to be very compelling— this really isn’t that. In fact, I don’t think there’s been a breaking change since the Py3 transition. I think PEP 563 was going to be the first — and that’s been delayed. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SZUL4ITKFUOYIJTAKVMBF4AYCKTTOKDQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Opt-in “iter def” and/or “gen def” syntax for generator functions
Thanks for your reply. > What's the advantage? I brought this up thinking about explicitness and readability. Say you want to figure out what this function is doing: def foo() -> t.Iterator[T]: [... 300 lines of code] ``` Is this a generator function? I'd argue that whether it's a generator function or not is fundamental to being able to read it. The type hint alone doesn't tell you whether you're looking at a generator function or not - it might just construct and return an iterator. So, you have to look for a `yield`. And if "yield" is somewhere in the function body, it will abruptly change the semantics of the entire definition. This feels a like spooky-action-at-a-distance to me - I'd much rather have the information up top in the function declaration, the same way that `async def` declares a coroutine function. However: this was actually discussed in PEP 255, where there was a decision *not* to introduce a new keyword for generator functions. From the BDFL decision: > No argument on either side is totally convincing, so I have consulted > my language designer’s intuition. It tells me that the syntax proposed > in the PEP is exactly right - not too hot, not too cold. But, like the Oracle > at Delphi in Greek mythology, it doesn’t tell me why, so I don’t have a > rebuttal for the arguments against the PEP syntax. - Aaron ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MVNK3UJHNFRPQZLHBMV47MLLT5P6OM25/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Python array multiplication for negative values should throw an error
Thank you, that is a good point! I would expect a similar error message for multiplication with negative integers. A real-world example could be the scenario where I encountered this: arrays were consistently assumed to be NumPy arrays, but in some cases cached versions of these arrays would be loaded from a JSON file without casting them to a NumPy array. Because the function signatures all state NumPy arrays, I applied an array * -1 operation before interpolation and other operations, in the expectation that this would yield the inverse of the elements instead of an empty list. It took me a while to find out that the problem was not in the interpolation or other operations following it, but in the array type. Of course, such bugs do not occur solely due to the lack of a raised errors on negative multiplication: in this case, a combination of a faulty assumption on the programmers' part and Python's lack of strict typing. However, raising an error on negative multiplication would immediately make it clear what is wrong, instead of hiding it. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/U4YXU3KIMCCZ4XFGSZVVMCBF6MJDNNEW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Python array multiplication for negative values should throw an error
Thanks for your response. While legacy production code is always an issue with potentially breaking changes, this line of thought would mean that future versions of Python could never introduce breaking changes. This should not be the case and is not the case now: for new Python versions, developers are expected to test and check for breaking changes, not just update production environments to a new version and hope for the best. The fact that this would throw an error instead of the expected empty list should then quickly lead to the right solution, whereas the other way around, when you expect an error but get an empty list, is hard to debug. That is a tradeoff with a clear winner in my opinion. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IZORCHR4TZLKNSTPM3SKHIVT2K34F5H3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Python array multiplication for negative values should throw an error
Here is another problem in this general area. Multiplying an array by a float already raises a type error. >>> []*0.0 Traceback (most recent call last): TypeError: can't multiply sequence by non-int of type 'float' This problem can arrse 'by accident'. For example, consider >>> x = {0.0} >>> x.update(range(3)) >>> list(x) [0.0, 1, 2] Let's now return to the feature (of the builtin list, tuple and str classes): >>> [1] * (-1) == [] True On the one hand I expect there's code that relies on the feature to run correctly. On the other hand I expect there's code that should raise an exception but doesn't, because of this feature. @ericfahlgren It would be most helpful if you could provide or create a reference to support your claim, that there is already a lot of code that would fail if this change was made. @fjwillemsen It would be most helpful if you could provide real-world examples of code where raising an exception would enable better code. Examples relating this are of course welcome from all, not just the two named contributors. -- Jonathan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6M6SRF6TGKR26BCCC2JIQEKFEFYLADZQ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Opt-in “iter def” and/or “gen def” syntax for generator functions
On Tue, 31 May 2022 at 23:00, Aaron L via Python-ideas wrote: > > After getting used to writing async functions, I’ve been wanting use a > similar syntax to declare generator functions. Something along the lines of > > `iter def my_iterator() -> T` > > and/or > > `gen def my_generator() -> (T, U, V)` > > Obviously, for backwards compatibility, this would need to be optional or > have an opt-in mechanism. Would a feature like this be at all within the > realm of possibility? I’d be happy to write up a much longer discussion if so. > > (I found a short discussion of such a feature in the archives about 8 years > ago[1]. But, since it predates both `async def` and the current type checker > regime, I thought it might be worth discussing. Apologies if I missed any > more recent discussions.) > What's the advantage? You can just use normal function syntax to define them, and it works correctly. Do you need the ability to declare that it's a generator even without any yields in it? ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DLMXTYBDTVXBMK2PCXIWGL7WDZT42ON2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Python array multiplication for negative values should throw an error
On Tue, May 31, 2022 at 5:54 AM fjwillemsen--- via Python-ideas < python-ideas@python.org> wrote: > I can not think of good reasons why Python array multiplication should not > throw an error for negative multipliers, because it is meaningless to have > array multiplication by negative value in the way it is intended in Python. > I can think of several reasons, but the big one is "because there is a lot of production code out there that depends on this behavior." Maybe if Python were adding this feature today, the implementers might take your suggestion to heart and raise on negative length array construction. But, it did not and now there is a lot of code that already exists and would fail if this were changed. Consider it water under the bridge and move on. Eric ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/NIK5WPJZVEPLL654ZY6NBH2WFP4I3EOJ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Opt-in “iter def” and/or “gen def” syntax for generator functions
After getting used to writing async functions, I’ve been wanting use a similar syntax to declare generator functions. Something along the lines of `iter def my_iterator() -> T` and/or `gen def my_generator() -> (T, U, V)` Obviously, for backwards compatibility, this would need to be optional or have an opt-in mechanism. Would a feature like this be at all within the realm of possibility? I’d be happy to write up a much longer discussion if so. (I found a short discussion of such a feature in the archives about 8 years ago[1]. But, since it predates both `async def` and the current type checker regime, I thought it might be worth discussing. Apologies if I missed any more recent discussions.) Thanks, Aaron [1] https://mail.python.org/archives/list/python-ideas@python.org/thread/OVIHVRKFUN4KMDTVSIAAN2CGR7VXFGQS/#GE6RDNWTR4PPKSMKSGMCFBFUJ42FWWV6 ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YGFRUHDAOXPHUKODV5VOPQZGJFLRBWLT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Provide an asynchronus equivalent to itertools.chain, and possibly more of itertools
I recently started to dive into asynchronous programming in Python. My program iterates over text from posts on a website and analyses it. I wanted to add the ability to asynchronously apply the same analysis to posts from multiple websites. It seemed likely to me that this would be something people commonly want to do, but I was surprised to find no built-in solution. There are a few relevant stack overflow posts: https://stackoverflow.com/questions/55299564/join-multiple-async-generators-in-python In which people suggest either using the library aiostream: https://aiostream.readthedocs.io/en/latest/operators.html#aiostream.stream.merge Or use an implementation based on `asyncio.Queue`. It seems to me that the aiostream solution goes beyond what I imagine most people would want to do. The solution using `asyncio.Queue` I think is fine, but is tricky to understand and a bit ugly to have to implement as a helper in every project looking to do this sort of thing. I propose there should be an asynchronous version of `itertools.chain` e.g. called `achain` which would allow users to write something like: ``` async for item in achain(async_gen1(), async_gen2(), async_gen3()): process(item) ``` I think it's very clear what this expression is doing and I can't imagine it's just me who'd find it useful. I have made an effort to implement at prototype version here: https://github.com/0Hughman0/achain/tree/no_context It works for the most basic case, but I imagine there's lots of nitty gritty regarding error handling etc. that I won't have done right. I'd love to hear others thoughts. Particularly whether you think this would be good to add to the standard library. It seems natural to me to try and make asynchronous compatible version of lots of the Python generators, e.g. `enumerate` and others from `itertools`. Many thanks, Hugh ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/Z6W5ZH234XPJVFPGGQU6CPRENPHO5YWP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Python array multiplication for negative values should throw an error
Hopefully this is not a duplicate of an existing thread or in the wrong section, first time posting here. In Python, array multiplication is quite useful to repeat an existing array, e.g. [1,2,3] * 2 becomes [1,2,3,4,5,6]. Multiplication by 0 yields an empty array: [1,2,3] * 0 becomes []. However, operations such as [numpy array] * -1 are very common to get the inverse of an array of numbers. The confusion here stems from the lack of type checking: while the programmer should check whether the array is a NumPy array or a Python array, this is not always done, giving rise to difficult to trace cases where [1,2,3] * -1 yields [] instead of [-1,-2,-3]. I can not think of good reasons why Python array multiplication should not throw an error for negative multipliers, because it is meaningless to have array multiplication by negative value in the way it is intended in Python. Instead I would propose that array multiplication by negative value throws an error. I would like to hear your opinions on this matter. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/47DZFIB46JQ55R7QFVVDDR4E2VLN2W3X/ Code of Conduct: http://python.org/psf/codeofconduct/