Re: NaN comparisons - Call For Anecdotes
On Fri, 18 Jul 2014 01:36:24 +1000, Chris Angelico wrote: On Fri, Jul 18, 2014 at 1:12 AM, Johann Hibschman jhibsch...@gmail.com wrote: Well, I just spotted this thread. An easy example is, well, pretty much any case where SQL NULL would be useful. Say I have lists of borrowers, the amount owed, and the amount they paid so far. nan = float(nan) borrowers = [Alice, Bob, Clem, Dan] amount_owed = [100.0, nan, 200.0, 300.0] amount_paid = [100.0, nan, nan, 200.0] who_paid_off = [b for (b, ao, ap) in zip(borrowers, amount_owed, amount_paid) if ao == ap] I want to just get Alice from that list, not Bob. I don't know how much Bow owes or how much he's paid, so I certainly don't know that he's paid off his loan. But you also don't know that he hasn't. NaN doesn't mean unknown, it means Not a Number. You need a more sophisticated system that allows for uncertainty in your data. I would advise using either None or a dedicated singleton (something like `unknown = object()` would work, or you could make a custom type with a more useful repr) Hmmm, there's something to what you say there, but IEEE-754 NANs seem to have been designed to do quadruple (at least!) duty with multiple meanings, including: - Missing values (I took a reading, but I can't read my handwriting). - Data known only qualitatively, not quantitatively (e.g. windspeed = fearsome). - Inapplicable values, e.g. the average depth of the oceans on Mars. - The result of calculations which are mathematically indeterminate, such as 0/0. - The result of real-valued calculations which are invalid due to domain errors, such as sqrt(-1) or acos(2.5). - The result of calculations which are conceptually valid, but are unknown due to limitations of floats, e.g. you have two finite quantities which have both overflowed to INF, the difference between them ought to be finite, but there's no way to tell what it should be. It seems to me that the way you treat a NAN will often depend on which category it falls under. E.g. when taking the average of a set of values, missing values ought to be skipped over, while actual indeterminate NANs ought to carry through: average([1, 1, 1, Missing, 1]) = 1 average([1, 1, 1, 0/0, 1]) = NAN I know that R distinguishes between NA and IEEE-754 NANs, although I'm not sure how complete its support for NANs is. But many (most?) R functions take an argument controlling whether or not to ignore NA values. In principle, you can encode the different meanings into NANs using the payload. There are 9007199254740988 possible Python float NANs. Half of these are signalling NANs, half are quiet NANs. Ignoring the sign bit leaves us with 2251799813685247 distinct sNANs and the same qNANs. That's enough to encode a *lot* of different meanings. [Aside: I find myself perplexed why IEEE-754 says that the sign bit of NANs should be ignored, but then specifies that another bit is to be used to distinguish signalling from quiet NANs. Why not just interpret NANs with the sign bit set are signalling, those with it clear are quiet?] -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Sat, Jul 19, 2014 at 3:57 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Hmmm, there's something to what you say there, but IEEE-754 NANs seem to have been designed to do quadruple (at least!) duty with multiple meanings, including: - Missing values (I took a reading, but I can't read my handwriting). - Data known only qualitatively, not quantitatively (e.g. windspeed = fearsome). - Inapplicable values, e.g. the average depth of the oceans on Mars. - The result of calculations which are mathematically indeterminate, such as 0/0. - The result of real-valued calculations which are invalid due to domain errors, such as sqrt(-1) or acos(2.5). - The result of calculations which are conceptually valid, but are unknown due to limitations of floats, e.g. you have two finite quantities which have both overflowed to INF, the difference between them ought to be finite, but there's no way to tell what it should be. Huh, okay. I thought the definition of NaN was based on the fourth one (mathematically indeterminate) and then it logically accepted the subsequent two (sqrt(-1) IMO is better handled by either a complex number or a thrown error, but NaN does make some sense there; definitely inf-inf = nan is as logical as 0/0 = nan). The first two seem to be better handled by SQL's NULL value (or non-value, or something, or maybe not something); the third is a bit trickier. Although the average of no values is logically calculated as 0/0 (ergo NaN makes sense there), I would say NaN isn't really right for a truly inapplicable value - for instance, recording the mass of a non-physical object. In an inventory system, it's probably simplest to use 0.0 to mean non-physical item, but it might be worth distinguishing between physical item with sufficiently low mass that it underflows our measurements (like a single sheet of paper when you're working with postal scales) and non-physical item with no meaningful mass (like credit card fees). In that case, I'm not sure that NaN is really appropriate to the situation, but would defer to IEE 754 on the subject. Obviously it's possible to abuse anything to mean anything (I do remember using nullable fields in DB2 to mean everything from inherit this value from parent to here be magic, code will work out the real value on the fly), but this is a question of intent and good design. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Anders J. Munch 2...@jmunch.dk writes: So far I received exactly the answer I was expecting. 0 examples of NaN!=NaN being beneficial. I wasn't asking for help, I was making a point. Whether that will lead to improvement of Python, well, I'm not too optimistic, but I feel the point was worth making regardless. Well, I just spotted this thread. An easy example is, well, pretty much any case where SQL NULL would be useful. Say I have lists of borrowers, the amount owed, and the amount they paid so far. nan = float(nan) borrowers = [Alice, Bob, Clem, Dan] amount_owed = [100.0, nan, 200.0, 300.0] amount_paid = [100.0, nan, nan, 200.0] who_paid_off = [b for (b, ao, ap) in zip(borrowers, amount_owed, amount_paid) if ao == ap] I want to just get Alice from that list, not Bob. I don't know how much Bow owes or how much he's paid, so I certainly don't know that he's paid off his loan. Cheers, Johann -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Fri, Jul 18, 2014 at 1:12 AM, Johann Hibschman jhibsch...@gmail.com wrote: Well, I just spotted this thread. An easy example is, well, pretty much any case where SQL NULL would be useful. Say I have lists of borrowers, the amount owed, and the amount they paid so far. nan = float(nan) borrowers = [Alice, Bob, Clem, Dan] amount_owed = [100.0, nan, 200.0, 300.0] amount_paid = [100.0, nan, nan, 200.0] who_paid_off = [b for (b, ao, ap) in zip(borrowers, amount_owed, amount_paid) if ao == ap] I want to just get Alice from that list, not Bob. I don't know how much Bow owes or how much he's paid, so I certainly don't know that he's paid off his loan. But you also don't know that he hasn't. NaN doesn't mean unknown, it means Not a Number. You need a more sophisticated system that allows for uncertainty in your data. I would advise using either None or a dedicated singleton (something like `unknown = object()` would work, or you could make a custom type with a more useful repr), and probably checking for it explicitly. It's entirely possible that you do virtually identical (or virtually converse) checks but with different handling of unknowns - for instance, you might have one check for who should be sent a loan reminder letter in which you leave out all unknowns, and another check for which accounts should be flagged for human checking in which you keep the unknowns (and maybe ignore every loan 100.0). You have a special business case here (the need to record information with a maybe state), and you need to cope with it, which means dedicated logic and planning and design and code. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Chris Angelico ros...@gmail.com writes: But you also don't know that he hasn't. NaN doesn't mean unknown, it means Not a Number. You need a more sophisticated system that allows for uncertainty in your data. Regardless of whether this is the right design, it's still an example of use. As to the design, using NaN to implement NA is a hack with a long history, see http://www.numpy.org/NA-overview.html for some color. Using NaN gets us a hardware-accelerated implementation with just about the right semantics. In a real example, these lists are numpy arrays with tens of millions of elements, so this isn't a trivial benefit. (Technically, that's what's in the database; a given analysis may look at a sample of 100k or so.) You have a special business case here (the need to record information with a maybe state), and you need to cope with it, which means dedicated logic and planning and design and code. Yes, in principle. In practice, everyone is used to the semantics of R-style missing data, which are reasonably well-matched by nan. In principle, (NA == 1.0) should be a NA (missing) truth value, as should (NA == NA), but in practice having it be False is more useful. As an example, indexing R vectors by a boolean vector containing NA yields NA results, which is a feature that I never want. Cheers, Johann -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Fri, Jul 18, 2014 at 4:49 AM, Johann Hibschman jhibsch...@gmail.com wrote: Chris Angelico ros...@gmail.com writes: But you also don't know that he hasn't. NaN doesn't mean unknown, it means Not a Number. You need a more sophisticated system that allows for uncertainty in your data. Regardless of whether this is the right design, it's still an example of use. Sure it is. And you may well have earned yourself that beer. But I don't put too much stock in hacks, at least as regards design decisions elsewhere. It's a little dubious when you grant special meaning to things and then use that meaning to justify the things' semantics. I'd much rather find an example where, for instance, numerical calculations might overflow to +inf or -inf, and then further calculations can result in a nan, etc, etc. Those are the sorts of examples that you'd find among SciPy users and such. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Fri, Jul 18, 2014 at 4:49 AM, Johann Hibschman jhibsch...@gmail.com wrote: In principle, (NA == 1.0) should be a NA (missing) truth value, as should (NA == NA), but in practice having it be False is more useful. This is actually fairly easily implemented, if you ever want it. class NAType: def __repr__(self): return NA def __eq__(self, other): return self __lt__ = __gt__ = __le__ = __ge__ = __ne__ = __eq__ NA = NAType() ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Chris Angelico ros...@gmail.com: numerical calculations might overflow to +inf or -inf, and then further calculations can result in a nan, etc, etc. Those are the sorts of examples that you'd find among SciPy users and such. There is some inconsistency. Mathematically, there are undefined operations, for a good reason. That's because the limits are not unambiguous and that's why 0/0, 0**0, 1/0 and inf-inf are undefined. Why 0/0 results in an exception but inf-inf = nan, I don't see a justification. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Thu, Jul 17, 2014 at 1:10 PM, Marko Rauhamaa ma...@pacujo.net wrote: Mathematically, there are undefined operations, for a good reason. That's because the limits are not unambiguous and that's why 0/0, 0**0, 1/0 and inf-inf are undefined. Well, 0**0 is usually defined as 1, despite the limits being ambiguous. Also, 1/0 in IEEE 754 is defined as inf. Why 0/0 results in an exception but inf-inf = nan, I don't see a justification. I expect that float division by zero was made to raise an exception for consistency with integer division by zero, where it might be considered inappropriate to switch types and return inf or nan. Granted that nowadays integer division returns a float anyway, but there is still floor division to think about. Maybe this should have been fixed in Python 3, but it wasn't. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Ian Kelly ian.g.ke...@gmail.com: Well, 0**0 is usually defined as 1, despite the limits being ambiguous. URL: https://www.math.hmc.edu/funfacts/ffiles/10005.3-5.shtml But if it could be defined, what should it be? 0 or 1? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Thu, Jul 17, 2014 at 3:08 PM, Marko Rauhamaa ma...@pacujo.net wrote: Ian Kelly ian.g.ke...@gmail.com: Well, 0**0 is usually defined as 1, despite the limits being ambiguous. URL: https://www.math.hmc.edu/funfacts/ffiles/10005.3-5.shtml But if it could be defined, what should it be? 0 or 1? I did say usually. There's not one single Holy Keeper of the Definitions for mathematics. Wikipedia lists some differing opinions on the subject: http://en.wikipedia.org/wiki/Exponentiation#Zero_to_the_power_of_zero I note that mathworld.wolfram.com also lists it as undefined, though. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Thu, Jul 17, 2014 at 5:00 PM, Ian Kelly ian.g.ke...@gmail.com wrote: On Thu, Jul 17, 2014 at 3:08 PM, Marko Rauhamaa ma...@pacujo.net wrote: Ian Kelly ian.g.ke...@gmail.com: Well, 0**0 is usually defined as 1, despite the limits being ambiguous. URL: https://www.math.hmc.edu/funfacts/ffiles/10005.3-5.shtml But if it could be defined, what should it be? 0 or 1? I did say usually. There's not one single Holy Keeper of the Definitions for mathematics. Wikipedia lists some differing opinions on the subject: http://en.wikipedia.org/wiki/Exponentiation#Zero_to_the_power_of_zero I note that mathworld.wolfram.com also lists it as undefined, though. Incidentally, as noted in the Wikipedia article you left out some options -- it's not just between 0 or 1. It's also possible to derive a limit of positive infinity or any nonnegative real. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Steven D'Aprano: I'll give you the benefit of the doubt, and assume that when you first posted you hadn't realised that the audience here does not have the relevant experience, but by refusing to ask the question elsewhere, and by making snide comments that they don't like beer, that pretty much gives the game away that you're looking to have your opinion confirmed rather than looking for an honest answer. I really did run out of time when I said I did: I spent the last 5 days running a chess tournament with 137 participants, with the tournament management software written in Python by yours truly. Which worked beautifully, Python is a great language. And I'm going to Iceland on vacation tomorrow, so I'm not going to start a new discussion of a different mailing list just now, maybe later. Not everyone is as prolific a writer as you are, Steven. The talk about the wrong list would have carried more weight if you had started out by saying wrong list, instead of going through every other argument first, before punting like that. By the way, which list is the appropriate one? The numpy and SciPy mailing lists are first and foremost about numpy and SciPy, I presume. Is there a general numerics-list somewhere also? I don't see any on https://mail.python.org/mailman/listinfo. By the way, you guys seem to have overlooked the opportunity for arbitrage: *Anyone* can go to a numerical list, poll for examples, and come back and earn some beer :) regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 15/07/2014 13:32, Anders J. Munch wrote: By the way, which list is the appropriate one? The numpy and SciPy mailing lists are first and foremost about numpy and SciPy, I presume. Is there a general numerics-list somewhere also? I don't see any on https://mail.python.org/mailman/listinfo. Try here http://news.gmane.org/index.php?prefix=gmane.comp.python -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
alister wrote: I don't have time to start this discussion over again on another mailing list. Don't anyone on those lists read python-list also? they possibly do, but prefer to keep discussions to the proper forum The semantics of the Python programming language is on-topic for python-list. This is about float.__eq__, not about numpy or SciPy. Maybe they just don't like beer? regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 14/07/2014 17:44, Anders J. Munch wrote: alister wrote: I don't have time to start this discussion over again on another mailing list. Don't anyone on those lists read python-list also? they possibly do, but prefer to keep discussions to the proper forum The semantics of the Python programming language is on-topic for python-list. This is about float.__eq__, not about numpy or SciPy. Maybe they just don't like beer? regards, Anders Maybe they recognise the eqivalent of our resident unicode expert. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Mon, Jul 14, 2014 at 10:44 AM, Anders J. Munch 2...@jmunch.dk wrote: alister wrote: I don't have time to start this discussion over again on another mailing list. Don't anyone on those lists read python-list also? they possibly do, but prefer to keep discussions to the proper forum The semantics of the Python programming language is on-topic for python-list. This is about float.__eq__, not about numpy or SciPy. Maybe they just don't like beer? Well, I don't see anybody with numerical expertise chiming in here to say they agree with you either, so that tells me you've probably got the wrong audience. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Mon, 14 Jul 2014 18:44:15 +0200, Anders J. Munch wrote: alister wrote: I don't have time to start this discussion over again on another mailing list. Don't anyone on those lists read python-list also? they possibly do, but prefer to keep discussions to the proper forum The semantics of the Python programming language is on-topic for python-list. This is about float.__eq__, not about numpy or SciPy. Maybe they just don't like beer? It's about NAN != NAN. The heaviest uses of NANs are numpy and scipy users. You remind me of the proverbial drunk who lost his keys in the park, but looks for them in the street under the street light because that's where the light is. That's assuming that you actually want an honest answer to your question. If you wanted to learn about use-cases for NANs, you ought to ask somewhere where there was a high proportion of numeric programmers, not on some list merely because it was convenient. I'll give you the benefit of the doubt, and assume that when you first posted you hadn't realised that the audience here does not have the relevant experience, but by refusing to ask the question elsewhere, and by making snide comments that they don't like beer, that pretty much gives the game away that you're looking to have your opinion confirmed rather than looking for an honest answer. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, 09 Jul 2014 23:09:36 +0200, Anders J. Munch wrote: Ethan Furman: I would suggest you ask for this on the numerical mailing lists instead of here -- and you may not want to offer a beer to everyone that has an anecdote for NaN behavior being useful. I don't have time to start this discussion over again on another mailing list. Don't anyone on those lists read python-list also? regards, Anders they possibly do, but prefer to keep discussions to the proper forum -- If you are going to walk on thin ice, you may as well dance. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 2014-07-10 01:57, Ben Finney wrote: Anders J. Munch 2...@jmunch.dk writes: Joel Goldstick wrote: I've been following along here, and it seems you haven't received the answer you want or need. So far I received exactly the answer I was expecting. 0 examples of NaN!=NaN being beneficial. Predictability and ease of diagnosis are the principles at work URL:http://stackoverflow.com/a/1573715/70157. You have already received examples of those. If those don't convince you of its usefulness, that's unfortunate, but at this point you are demonstrating a standard which is both unreasonably high (even the rationale of the committee doesn't convince you) and unreasonably low (you ask not for explanations but personal anecdotes). Good luck to you in your quest. I can think of one place where equality of NaNs would be useful: sorting. However, in that use-case, you would also want it to be orderable, perhaps greater than any other non-NaN float. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Thu, Jul 10, 2014 at 9:28 PM, MRAB pyt...@mrabarnett.plus.com wrote: I can think of one place where equality of NaNs would be useful: sorting. However, in that use-case, you would also want it to be orderable, perhaps greater than any other non-NaN float. In that case, you're setting special rules, so you probably want a key-based sort. Since the sorting of tuples works by comparing elements sequentially, it's simple enough: l=[1.0,2.0,1.5,float(nan),1.7] sorted(l,key=lambda n: (n!=n, n)) [1.0, 1.5, 1.7, 2.0, nan] I'm not sure what's supposed to happen if you just let sorted() go with its default comparisons. It seems to put the nan at the end, but I wouldn't bet on that being reliable, unless it's documented somewhere. Of course, with a key sort you can choose to put NaNs at the beginning just as easily: sorted(l,key=lambda n: (n==n, n)) [nan, 1.0, 1.5, 1.7, 2.0] You want custom rules for sorting? You got 'em. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, 09 Jul 2014 00:57:03 -0400, Terry Reedy wrote:
On 7/8/2014 8:10 PM, Steven D'Aprano wrote:
There's no force of law that requires Python to enforce reflexivity on
values where reflexivity does not apply,
There are laws of logic that make the lack of reflexivity obnoxious when
putting objects in collections.
Floating point numbers are not real numbers and do not obey the laws of
real arithmetic. Once you allow floats, or comparisons between arbitrary
types, no matter what you do, something is going to break and somebody is
going to be upset.
Python evaded the problem, at least for
some builtins, by contradicting itself and treating nans as equal to
themselves in the context of collections.
It's not just NANs. It's any non-reflexive type:
py class AlwaysUnequal:
... def __eq__(self, other): return False
...
py x = AlwaysUnequal()
py x in [1, 2, x]
True
py any(x == a for a in [1, 2, x])
False
In 2.x, 'in' was defined in terms of ==, but
nan=float('nan')
nl = [nan]
nan in nl
True
even though nan != the only member of nl.
In 3.x, 'in' was redefined to include 'is' as well as '=='.
I don't understand this. You've just shown an example from Python 2 where
'in' uses 'is'. How is that a Python 3 change?
--
Steve
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 4:43 PM, Steven D'Aprano st...@pearwood.info wrote: I don't understand this. You've just shown an example from Python 2 where 'in' uses 'is'. How is that a Python 3 change? A docs change. https://docs.python.org/2/reference/expressions.html#not-in For the list and tuple types, x in y is true if and only if there exists an index i such that x == y[i] is true. https://docs.python.org/3/reference/expressions.html#not-in For container types such as list, tuple, set, frozenset, dict, or collections.deque, the expression x in y is equivalent to any(x is e or x == e for e in y). So the definition of the operator (as stated in the docs) has changed to match its behaviour. That's definitely a change; the Py2 behaviour is buggy according to its docs. Should the Py2 docs be updated to match the Py3 docs, or should it be left as a minor technical distinction that almost never is important? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
I wrote: | class Monitor(Thread): | def run(self): | old = self.get_current_value() | while not self.Terminated: | new = self.get_current_value() | if new != old: | print(time.asctime(), changed to, new) | old = new | time.sleep(1) Huh, I don't know what happened to the identation here, I'l try again: class Monitor(Thread): def run(self): old = self.get_current_value() while not self.Terminated: new = self.get_current_value() if new != old: print(time.asctime(), changed to, new) old = new time.sleep(1) regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Steven D'Aprano wrote: It seems to me that the trivial work-around is: * gather packed floats from some device, as ints * process them *as ints* in some way which requires reflexivity * unpack back into floats * (maybe) much later perform numeric calculations on them Although perhaps I don't understand your use-case. Clearly you do not. floats are not ints. I have no idea how you imagine processing IEEE-754 floating-point values in int form. My use case is: Working with IEEE-754 floating-point values. That means storing and retrieving them, serialising and transferring them, accepting them as user input, printing them, all the usual things you do with values. And doing so in a way that does not require special handling in algorithms that are otherwise generic. When the same algorithm is capable of dealing with ints, bytestrings, text string, tuples, list, dictionaries, time stamps, NoneType's, bools, floating-point floats and a thousand other things, then NaNs stand out as the values that have special algorithm-breaking magic. I gave an example of such an algorithm in an earlier reply to Chris. regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Thu, Jul 10, 2014 at 1:08 AM, Anders J. Munch 2...@jmunch.dk wrote: Steven D'Aprano wrote: It seems to me that the trivial work-around is: * gather packed floats from some device, as ints * process them *as ints* in some way which requires reflexivity * unpack back into floats * (maybe) much later perform numeric calculations on them Although perhaps I don't understand your use-case. Clearly you do not. floats are not ints. I have no idea how you imagine processing IEEE-754 floating-point values in int form. My use case is: Working with IEEE-754 floating-point values. That means storing and retrieving them, serialising and transferring them, accepting them as user input, printing them, all the usual things you do with values. And doing so in a way that does not require special handling in algorithms that are otherwise generic. When the same algorithm is capable of dealing with ints, bytestrings, text string, tuples, list, dictionaries, time stamps, NoneType's, bools, floating-point floats and a thousand other things, then NaNs stand out as the values that have special algorithm-breaking magic. I gave an example of such an algorithm in an earlier reply to Chris. If you need to do bitwise comparisons, then the easiest way is to use the bitpattern, converted to an integer. A 64-bit float becomes a 64-bit integer. It's then very simple to compare them, and reflexivity is maintained. At what point do you actually need them to be floats? What are you really doing with them? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Chris Angelico: If you need to do bitwise comparisons, then the easiest way is to use the bitpattern, converted to an integer. A 64-bit float becomes a 64-bit integer. It's then very simple to compare them, and reflexivity is maintained. At what point do you actually need them to be floats? What are you really doing with them? What does one do with floats? Add, subtract, multipy, divide, display, input, store and retrieve to and from various formats. All the usual stuff. Why would my use be different from anyone elses? What you and Steven seem to be saying is that I should employ strategies to avoid NaNs ever being compared. I'll take that one step further and say that as long as NaN!=NaN, everyone should seek to avoid NaNs ever being compared. regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 07/08/2014 07:53 AM, Anders J. Munch wrote: So I make this claim: float.__eq__ implementing IEEE-754 NaN comparison rules creates real problems for developers. And it has never, ever, helped anyone do anything. Never is a strong claim, and easily disproven if false: Simply provide a counterexample. So that is my challenge: If you have a program (a pre-existing and useful one, not something artificial created for this challenge) that benefits from NaN!=NaN and that would fail if x==x for all float objects x, then please come forward and show it, and I'll buy you a beer the next time I'm at PyCon. I would suggest you ask for this on the numerical mailing lists instead of here -- and you may not want to offer a beer to everyone that has an anecdote for NaN behavior being useful. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, 09 Jul 2014 17:08:15 +0200, Anders J. Munch wrote: Steven D'Aprano wrote: It seems to me that the trivial work-around is: * gather packed floats from some device, as ints * process them *as ints* in some way which requires reflexivity * unpack back into floats * (maybe) much later perform numeric calculations on them Although perhaps I don't understand your use-case. Clearly you do not. floats are not ints. Of course not. But there is a one-to-one correspondence between a 64-bit float and a 64-bit int, and the conversion is lossless in both directions. When you talked about: So when my software reads these values in binary, unpack them using the struct module, and goes to work. I assumed that you realised that the 64-bit(?) values you were receiving in binary could be interpreted as ints. After all, you have to unpack them from some bytes. Since that's not what you're doing, I have no idea what it is. I have no idea how you imagine processing IEEE-754 floating-point values in int form. Cast your 64-bit float into a 64-bit int. Or, if it's a C single rather than a double, cast the 32-bit float into a 32-bit int. Now you can compare them for equality without carrying about NANs, and without losing data. Later, when you're ready to start doing some numeric work on them, you cast back to floats. That's the idea I had in mind. Perhaps it doesn't match your use-case. My use case is: Working with IEEE-754 floating-point values. That means storing and retrieving them, serialising and transferring them, accepting them as user input, printing them, all the usual things you do with values. But apparently not arithmetic? And doing so in a way that does not require special handling in algorithms that are otherwise generic. Ah, well there's your problem. NANs are special, as a signed zeroes and INFs. Does it distress you that x + x = x when x is an INF? When the same algorithm is capable of dealing with ints, bytestrings, text string, tuples, list, dictionaries, time stamps, NoneType's, bools, floating-point floats and a thousand other things, ^ Obviously not, or you wouldn't be complaining about the inability to handle floats. The hardware devices generating your float data... do they also generate ints, bytestrings, text strings, tuples, lists, dicts, time stamps, None, bools, and a thousand other things? If not, I wonder why you are insisting that you have to handle a *specialised* data type using a *generic* algorithm. then NaNs stand out as the values that have special algorithm-breaking magic. Well yes. Floats have all sorts of problems. They're horrible really, the worst possible way to model real numbers, except for all the other ways. I'm not unsympathetic to your problem, which is why I proposed two new operators, === and !==, and a change to == and !=, in another thread. Would == always doing an identity test before calling __eq__ solve your problem? If not, what would it take to solve your problem? -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 10:53 AM, Steven D'Aprano
Cast your 64-bit float into a 64-bit int. Or, if it's a C single rather
than a double, cast the 32-bit float into a 32-bit int. Now you can
compare them for equality without carrying about NANs, and without losing
data. Later, when you're ready to start doing some numeric work on them,
you cast back to floats. That's the idea I had in mind. Perhaps it
doesn't match your use-case.
Unfortunately, it's not that simple:
import math, struct
def float_to_int(x):
... return struct.unpack('L', struct.pack('d', x))[0]
...
0.0 == -0.0
True
float_to_int(0.0) == float_to_int(-0.0)
False
nan1 = struct.unpack('d', b'\x00'*6+b'\xf1\x7f')[0]
math.isnan(nan1)
True
nan2 = struct.unpack('d', b'\x00'*6+b'\xf2\x7f')[0]
math.isnan(nan2)
True
float_to_int(nan1) == float_to_int(nan1)
True
float_to_int(nan2) == float_to_int(nan2)
True
float_to_int(nan1) == float_to_int(nan2)
False
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Steven D'Aprano wrote: I assumed that you realised that the 64-bit(?) values you were receiving in binary could be interpreted as ints. After all, you have to unpack them from some bytes. Since that's not what you're doing, I have no idea what it is. Stop obsessing over how NaN's came to exist in my software. That's just context. The argument is over how those NaNs should behave. Their provenance is not relevant. I have no idea how you imagine processing IEEE-754 floating-point values in int form. Cast your 64-bit float into a 64-bit int. I can construct a bijective mapping between any data structure and a subset of the natural numbers, but that has nothing to do with practical programming. A cast value would be impossible to work with. My use case is: Working with IEEE-754 floating-point values. That means storing and retrieving them, serialising and transferring them, accepting them as user input, printing them, all the usual things you do with values. But apparently not arithmetic? Of course also arithmetic. I left it out of the list because then you would say hah! if you're doing arithmetic then it's not a generic algorithm. Apparently I can't win, you are going to nitpick anything I write. Ah, well there's your problem. NANs are special, as a signed zeroes and INFs. Does it distress you that x + x = x when x is an INF? No. When the same algorithm is capable of dealing with ints, bytestrings, text string, tuples, list, dictionaries, time stamps, NoneType's, bools, floating-point floats and a thousand other things, ^ Obviously not, or you wouldn't be complaining about the inability to handle floats. NaNs are not floating-point values. A floating-point value has a sign, an exponent and a mantissa. They are IEEE 754 floating point values, though. The hardware devices generating your float data... do they also generate ints, bytestrings, text strings, tuples, lists, dicts, time stamps, None, bools, and a thousand other things? If not, I wonder why you are insisting that you have to handle a *specialised* data type using a *generic* algorithm. All the other types are also specialised, for their separate purpose. That doesn't make them non-reflexive. I'm not unsympathetic to your problem, which is why I proposed two new operators, === and !==, and a change to == and !=, in another thread. Would == always doing an identity test before calling __eq__ solve your problem? If not, what would it take to solve your problem? It would not solve it. Two bitwise identical NaNs would still compare different. What would solve the problem is making identical NaNs compare equal. regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Ethan Furman: I would suggest you ask for this on the numerical mailing lists instead of here -- and you may not want to offer a beer to everyone that has an anecdote for NaN behavior being useful. I don't have time to start this discussion over again on another mailing list. Don't anyone on those lists read python-list also? regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 5:09 PM, Anders J. Munch 2...@jmunch.dk wrote: Ethan Furman: I would suggest you ask for this on the numerical mailing lists instead of here -- and you may not want to offer a beer to everyone that has an anecdote for NaN behavior being useful. I don't have time to start this discussion over again on another mailing list. Don't anyone on those lists read python-list also? I've been following along here, and it seems you haven't received the answer you want or need. Since you are the one in search of a solution it seems that you might take more kindly to a pointer in a direction that might serve you better. Best of luck regards, Anders -- https://mail.python.org/mailman/listinfo/python-list -- Joel Goldstick http://joelgoldstick.com -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 07/09/2014 02:09 PM, Anders J. Munch wrote: Ethan Furman: I would suggest you ask for this on the numerical mailing lists instead of here -- and you may not want to offer a beer to everyone that has an anecdote for NaN behavior being useful. I don't have time to start this discussion over again on another mailing list. Don't anyone on those lists read python-list also? I believe there are a handful, but I don't know how active they are here. For a straight-up numbers type question, especially about NaN and who's using it to what effect, a numbers or scientific mailing list is really the way to go. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Joel Goldstick wrote: I've been following along here, and it seems you haven't received the answer you want or need. So far I received exactly the answer I was expecting. 0 examples of NaN!=NaN being beneficial. I wasn't asking for help, I was making a point. Whether that will lead to improvement of Python, well, I'm not too optimistic, but I feel the point was worth making regardless. regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 5:03 PM, Anders J. Munch 2...@jmunch.dk wrote: Joel Goldstick wrote: I've been following along here, and it seems you haven't received the answer you want or need. So far I received exactly the answer I was expecting. 0 examples of NaN!=NaN being beneficial. I wasn't asking for help, I was making a point. Whether that will lead to improvement of Python, well, I'm not too optimistic, but I feel the point was worth making regardless. If the reason there were 0 examples is because the group that you posed the question to is lacking the kind of experience needed to be able to provide them (speaking for myself, I wouldn't know because I don't ever do anything with NaN beyond checking for it as an error condition), then the point hasn't really been made, has it? -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 07/09/2014 04:03 PM, Anders J. Munch wrote: Joel Goldstick wrote: I've been following along here, and it seems you haven't received the answer you want or need. So far I received exactly the answer I was expecting. 0 examples of NaN!=NaN being beneficial. python-list (in whichever from you interact with it) is in no way specialized, yet you are asking for a highly specialized bit of information. I wasn't asking for help, I was making a point. Whether that will lead to improvement of Python, well, I'm not too optimistic, but I feel the point was worth making regardless. The only point you have made is that no-one here uses them much. You could just as easily ask how many of us use lead-alloy to shield our in-home nuclear reactors to get the same answer. On the other hand, if you go and ask on the numpy and simpy and whatever-math-or-science-py lists there are, and get affirmative answers from them that yeah, NaN != NaN is just a huge pain and everyone is working around it, then brought that information back here -- well, then you will have made the point that you want to make. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Anders J. Munch 2...@jmunch.dk writes: Joel Goldstick wrote: I've been following along here, and it seems you haven't received the answer you want or need. So far I received exactly the answer I was expecting. 0 examples of NaN!=NaN being beneficial. Predictability and ease of diagnosis are the principles at work URL:http://stackoverflow.com/a/1573715/70157. You have already received examples of those. If those don't convince you of its usefulness, that's unfortunate, but at this point you are demonstrating a standard which is both unreasonably high (even the rationale of the committee doesn't convince you) and unreasonably low (you ask not for explanations but personal anecdotes). Good luck to you in your quest. -- \ “Every man would like to be God, if it were possible; some few | `\ find it difficult to admit the impossibility.” —Bertrand | _o__)Russell, _Power: A New Social Analysis_, 1938 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tuesday, July 8, 2014 8:23:47 PM UTC+5:30, Anders J. Munch wrote: Most people don't need to deal with NaN's in Python at all, fortunately. They just don't appear in normal computation, because the interpreter raises an exception instead. So I make this claim: float.__eq__ implementing IEEE-754 NaN comparison rules creates real problems for developers. And it has never, ever, helped anyone do anything. Never is a strong claim, and easily disproven if false: Simply provide a counterexample. So that is my challenge: If you have a No I dont have a direct answer for your challenge but I have two close analogies. Nan in floats is analogous (almost isomorphic) to two other areas - null in RDBMS - bottom – ⊥ – in denotational semantics The denotational semantics example would really clarify the issue. I have discussed it here in the haskell context: http://blog.languager.org/2012/08/functional-programming-philosophical.html http://en.wikibooks.org/wiki/Haskell/Denotational_semantics#.E2.8A.A5_Bottom is a more general survey RDBMS nulls is probably a more familiar example. Lets say you have a dbms of people and one nullable field is 'telephone'. Now some people have no phones and the original intent of the nullable field was to take care of that. But then in data entry some people who had too many phones and could not decide which to fill kept the field null!! Are these two cases equivalent? Your request amounts to making them so. Now read your own case: It happens in my work I come across them quite a lot. I'm writing software that talks to embedded applications that can contain NaN values for a variety of reasons - never-initialised storage, initialise-to-NaN, hardware failures etc. How do you decide that all these are equivalent/ 'equal'? Someone or other somewhere or other will be unhappy!! My analysis of the problem: The *letter* of the IEEE standard describes nans (and other exotic beasties) The *spirit* describes exceptional conditions that arise in floating computations [I have not read the standard -- just guessing] The standard was made when programming languages did not have well tried and tested exception mechanisms. So with python supporting nans we have two competing exception mechanisms: - the IEEE one -- nans - the native python one -- exceptions Once this is clear it should be clear what the solution should be: DRY, no redundancy etc. Since exceptions are pythonic, nans should not be supported. If we were starting from scratch this would be the way to go. However given the currently extant support, this would be a downgrade for people like you -- who would be unhappy! So intermediate more kludgy solution: The problem is not with nan==nan False But with type(nan) class 'float' So nan is a float but by definition is not a number!! So float a superset of all numbers??!! If instead like: type(None) class 'NoneType' we had type(None) class 'NanType' then normal users would never see it and people like you could deal with it outside the float-umbrella. -- https://mail.python.org/mailman/listinfo/python-list
NaN comparisons - Call For Anecdotes
Most people don't need to deal with NaN's in Python at all, fortunately. They just don't appear in normal computation, because the interpreter raises an exception instead. It happens in my work I come across them quite a lot. I'm writing software that talks to embedded applications that can contain NaN values for a variety of reasons - never-initialised storage, initialise-to-NaN, hardware failures etc. So when my software reads these values in binary, unpack them using the struct module, and goes to work. And NaN's are no different from any other value, it's something to store, compare, display etc. And that worked fine in my Python 2.4 apps. Then I upgraded to 2.7 and it broke. Because 2.7 goes out of it's way to ensure that NaN's don't compare equal to themselves. I discovered it when a sanity check told me that two functions, to_binary and from_binary, weren't each other's inverse, as they were intended to be. Apparently, bitPattern==to_binary(from_binary(bitPattern)) wasn't true for this particular value of bitPattern. Of course, the bit pattern in question was the binary representation for a floating-point NaN. Panic time! If I can't trust == to return True for (mathematically) equal objects, that means that every single algorithm I had ever written that explicitly or implicitly does .__eq__ or .__ne__ comparison was suspect! That meant I had 3 lines of code to review. Every time there's a comparison, if there was any chance that either value could be a float NaN, I would have to change e.g. if x==y: to if x==y or (isinstance(x, float) and isinstance(y, float) and math.isnan(x) and math.isnan(y)): To make it bearable, I could wrap the pattern up in a function and write if my_equal(x,y): but I would still lose, because the standard library does == and != all over the place without calling my_equal. In the end I came up with this hack: Every time I struct.unpack'd a float, I check if it's a NaN, and if it is, then I replace it with a reference to a single, shared, canonical NaN. That means that container objects that skip __equal__ when comparing an object to itself will work -- e.g. hash keys. It's half a solution, of course: Any further computation with a NaN value will change it to a different NaN object, so I still needed to do explicit NaN-checks in various places. I'm sure there are still NaN-related bugs in my code, but right now it's good enough - I haven't seen NaN-related bugs in a good while. Now, all this bothers me. Not that I had to do some work to get stuff to work in an imperfect world. No, what bothers me is that this behaviour was explicitly and deliberately put in for no good reason. The only reason is standard says so. Not that there's any legal requirement for Python to follow the IEEE-754 standard. Or for that matter, for Python's spelling of IEEE-754 comparisons to be ==. So I make this claim: float.__eq__ implementing IEEE-754 NaN comparison rules creates real problems for developers. And it has never, ever, helped anyone do anything. Never is a strong claim, and easily disproven if false: Simply provide a counterexample. So that is my challenge: If you have a program (a pre-existing and useful one, not something artificial created for this challenge) that benefits from NaN!=NaN and that would fail if x==x for all float objects x, then please come forward and show it, and I'll buy you a beer the next time I'm at PyCon. regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 12:53 AM, Anders J. Munch 2...@jmunch.dk wrote:
In the end I came up with this hack: Every time I struct.unpack'd a
float, I check if it's a NaN, and if it is, then I replace it with a
reference to a single, shared, canonical NaN. That means that
container objects that skip __equal__ when comparing an object to
itself will work -- e.g. hash keys.
Let's take a step back.
No, let's take a step forward.
And let's take a step back again.
(And we're building a military-grade laser!)
Why *should* all NaNs be equal to each other? You said on the other
list that NaN==NaN was equivalent to (2+2)==(1+3), but that assumes
that NaN is a single thing. It's really describing the whole huge
area of stuff that just ain't numbers. Imagine if (x + y) wasn't 4,
but was table. And (a + b) turned out to be cyan. Does table equal
cyan, just because neither of them is a number? Certainly not. Neither
should (inf - inf) be equal to (inf / inf). Both of those expressions
evaluate to something that can't possibly be a number - it can't be
anywhere on the number line, it can't be anywhere on the complex
plane, it simply isn't a number. And they're not the same non-numeric
thing.
For hash keys, float object identity will successfully look them up:
d={}
d[float(nan)]=1
d[float(nan)]=2
x=float(nan)
d[x]=3
d[x]
3
d
{nan: 1, nan: 2, nan: 3}
So I'm not sure where the problems come from. You can iterate over a
dict's keys and look things up with them:
for k,v in d.items():
print(k,v,d[k])
nan 1 1
nan 2 2
nan 3 3
You can do a simple 'is' check as well as your equality check:
if x is y or x == y:
print(They're the same)
But any time you compare floats for equality, you *already* have to
understand what you're doing (because of rounding and such), so I
don't see why the special case on NaN is significant, unless as
mentioned above, you want all NaNs to be equal, which doesn't make
sense.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, Jul 8, 2014 at 8:53 AM, Anders J. Munch 2...@jmunch.dk wrote: And that worked fine in my Python 2.4 apps. Then I upgraded to 2.7 and it broke. Because 2.7 goes out of it's way to ensure that NaN's don't compare equal to themselves. As far as I know nothing changed between 2.4 and 2.7 in this regard. Python has always had NaN compare unequal to everything, per the standard. I discovered it when a sanity check told me that two functions, to_binary and from_binary, weren't each other's inverse, as they were intended to be. Apparently, bitPattern==to_binary(from_binary(bitPattern)) wasn't true for this particular value of bitPattern. Of course, the bit pattern in question was the binary representation for a floating-point NaN. Okay, here's your problem: there isn't just one binary representation for NaN. The standard defines any value with all 1's in the exponent and a non-zero significand as NaN (a zero significand would instead be an infinity). Your bit comparison is going to have to be prepared to compare NaNs that don't have the same binary representation. By the way, there are also multiple binary representations for 0. If you compare them as floats, then they'll compare equal to one another, but if you're just comparing binary representations then you'll have issues there as well. Now, all this bothers me. Not that I had to do some work to get stuff to work in an imperfect world. No, what bothers me is that this behaviour was explicitly and deliberately put in for no good reason. The only reason is standard says so. Not that there's any legal requirement for Python to follow the IEEE-754 standard. Or for that matter, for Python's spelling of IEEE-754 comparisons to be ==. Following the standard isn't a good reason itself? It seems to me that one should be expected to provide a strong justification for *deviating* from the standard, not for following it. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, Jul 8, 2014 at 10:19 AM, Chris Angelico ros...@gmail.com wrote:
For hash keys, float object identity will successfully look them up:
d={}
d[float(nan)]=1
d[float(nan)]=2
x=float(nan)
d[x]=3
d[x]
3
d
{nan: 1, nan: 2, nan: 3}
Neat!
S
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 1:24 AM, Skip Montanaro s...@pobox.com wrote:
On Tue, Jul 8, 2014 at 10:19 AM, Chris Angelico ros...@gmail.com wrote:
For hash keys, float object identity will successfully look them up:
d={}
d[float(nan)]=1
d[float(nan)]=2
x=float(nan)
d[x]=3
d[x]
3
d
{nan: 1, nan: 2, nan: 3}
Neat!
Yeah. It's one of those arguable points; is it a mere optimization
that dict lookup does an identity check before an equality check, or
is it crucial to other invariants (like the iteration one - if you
iterate over items(), it should give exactly the same results as
iterating over keys() and then looking things up)? Obviously it's
better for the dict to use equality checks rather than identity checks
(otherwise, at the very least, you'd have to explicitly intern all
strings used as dict keys - that'd just be ridiculous), but with
objects that don't compare equal to themselves, what should be done?
I think Python's picked a quite reasonable approach.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Chris Angelico ros...@gmail.com: Why *should* all NaNs be equal to each other? I appreciate why you can't say NaN is equal to NaN. However, shouldn't the very comparison attempt trigger an arithmetic exception? After all, so does a division by zero. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 2:16 AM, Marko Rauhamaa ma...@pacujo.net wrote: Chris Angelico ros...@gmail.com: Why *should* all NaNs be equal to each other? I appreciate why you can't say NaN is equal to NaN. However, shouldn't the very comparison attempt trigger an arithmetic exception? After all, so does a division by zero. I'd say it would surprise people rather a lot if operations like dict insertion/lookup could trigger arithmetic exceptions. :) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, 08 Jul 2014 16:53:47 +0200, Anders J. Munch wrote:
Most people don't need to deal with NaN's in Python at all, fortunately.
They just don't appear in normal computation, because the interpreter
raises an exception instead.
It happens in my work I come across them quite a lot. I'm writing
software that talks to embedded applications that can contain NaN values
for a variety of reasons - never-initialised storage, initialise-to-NaN,
hardware failures etc.
So when my software reads these values in binary, unpack them using the
struct module, and goes to work. And NaN's are no different from any
other value, it's something to store, compare, display etc.
And that worked fine in my Python 2.4 apps.
I think you're smoking something funny :-)
[steve@ando ~]$ python2.4
Python 2.4.3 (#1, Jan 9 2013, 06:49:54)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2
Type help, copyright, credits or license for more information.
py NAN = float('NAN')
py NAN == NAN
False
NANs compared unequal in Python 2.4.
[...]
Now, all this bothers me. Not that I had to do some work to get stuff
to work in an imperfect world. No, what bothers me is that this
behaviour was explicitly and deliberately put in for no good reason.
Oh, you've read the IEEE-754 standard, and that's what it says? We're
going to specify this behaviour for NANs just to annoy people perhaps?
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 3:13 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Now, all this bothers me. Not that I had to do some work to get stuff to work in an imperfect world. No, what bothers me is that this behaviour was explicitly and deliberately put in for no good reason. Oh, you've read the IEEE-754 standard, and that's what it says? We're going to specify this behaviour for NANs just to annoy people perhaps? You're confusing IEEE with Adobe. IEEE is pronounced Ai!, but it's Adobe that makes people scream that. http://www.theregister.co.uk/2009/04/30/xee_photoshop_psd_code_rant/ ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, Jul 8, 2014 at 9:53 AM, Anders J. Munch 2...@jmunch.dk wrote: Most people don't need to deal with NaN's in Python at all, fortunately. They just don't appear in normal computation, because the interpreter raises an exception instead. In addition to what others have written, I will add one thing. There are certainly situations where raising an exception is bad. Consider all the people in the scientific computing community doing fancy linear algebra sorts of things, often with missing data. They generally want NaN propagated and not have some long running calculation crash in the middle. Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Chris Angelico ros...@gmail.com: I'd say it would surprise people rather a lot if operations like dict insertion/lookup could trigger arithmetic exceptions. :) That wouldn't trigger exceptions. Dict operations do an is test before an == test. In fact, you couldn't even use NaN as a dict key otherwise. Thus, dict operations never test NaN == NaN. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Skip Montanaro s...@pobox.com: In addition to what others have written, I will add one thing. There are certainly situations where raising an exception is bad. Consider all the people in the scientific computing community doing fancy linear algebra sorts of things, often with missing data. They generally want NaN propagated and not have some long running calculation crash in the middle. Do the scientific computers mind: 1 / 0 Traceback (most recent call last): File stdin, line 1, in module ZeroDivisionError: division by zero or would they prefer their fancy linear-algebraic computation to just forge on? Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
import numpy a1 = numpy.ones(5) a1 array([ 1., 1., 1., 1., 1.]) a0 = numpy.zeros(5) a0 array([ 0., 0., 0., 0., 0.]) a1 / a0 __main__:1: RuntimeWarning: divide by zero encountered in true_divide array([ inf, inf, inf, inf, inf]) nans = numpy.array([float(nan)] * 5) nans array([ nan, nan, nan, nan, nan]) a1 / nans array([ nan, nan, nan, nan, nan]) a1 / a0 * nans array([ nan, nan, nan, nan, nan]) You get a runtime warning (this is in Python 2.7), but the division returns the appropriate value, in this case, infinity. So, yes, they forge on, and NaN taints things just about the way you'd expect. Skip On Tue, Jul 8, 2014 at 12:36 PM, Marko Rauhamaa ma...@pacujo.net wrote: Skip Montanaro s...@pobox.com: In addition to what others have written, I will add one thing. There are certainly situations where raising an exception is bad. Consider all the people in the scientific computing community doing fancy linear algebra sorts of things, often with missing data. They generally want NaN propagated and not have some long running calculation crash in the middle. Do the scientific computers mind: 1 / 0 Traceback (most recent call last): File stdin, line 1, in module ZeroDivisionError: division by zero or would they prefer their fancy linear-algebraic computation to just forge on? Marko -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 3:31 AM, Marko Rauhamaa ma...@pacujo.net wrote: Chris Angelico ros...@gmail.com: I'd say it would surprise people rather a lot if operations like dict insertion/lookup could trigger arithmetic exceptions. :) That wouldn't trigger exceptions. Dict operations do an is test before an == test. In fact, you couldn't even use NaN as a dict key otherwise. Thus, dict operations never test NaN == NaN. Check out the example I posted early in this thread of a dict with three keys, all of them NaN. And note that hash(float(nan))==0. Now try looking up d[0]. Before it raises KeyError, it has to compare that 0 for equality with each of the nans, because it can't shortcut it based on the hash. In fact, I can prove it thus: class X: def __eq__(self, other): if self is other: print(Comparing against self - I am me!) return True print(Comparing against,other,-,id(other)) return False def __hash__(self): return 0 d[X()] Comparing against nan - 18777952 Comparing against nan - 19624864 Comparing against nan - 18776272 Traceback (most recent call last): File pyshell#20, line 1, in module d[X()] KeyError: __main__.X object at 0x016B40D0 Any lookup of anything with a hash of 0 will do this. 0 itself (as any type of number), another NaN, or anything at all. For the dict to work sanely, these comparisons have to work and be False. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 3:54 AM, Chris Angelico ros...@gmail.com wrote: class X: def __eq__(self, other): if self is other: print(Comparing against self - I am me!) return True print(Comparing against,other,-,id(other)) return False def __hash__(self): return 0 d[X()] Comparing against nan - 18777952 Comparing against nan - 19624864 Comparing against nan - 18776272 Traceback (most recent call last): File pyshell#20, line 1, in module d[X()] KeyError: __main__.X object at 0x016B40D0 Better example: Subclass float, then it can actually *be* a nan. class NoisyFloat(float): def __eq__(self, other): print(Comparing,id(self),against,id(other)) return super().__eq__(other) def __hash__(self): return super().__hash__() d[NoisyFloat(nan)] Comparing 23777152 against 18777952 Comparing 23777152 against 19624864 Comparing 23777152 against 18776272 Traceback (most recent call last): File pyshell#35, line 1, in module d[NoisyFloat(nan)] KeyError: nan That's comparing nan==nan three types with four different nans. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 07/08/2014 07:53 AM, Anders J. Munch wrote:
Most people don't need to deal with NaN's in Python at all,
fortunately. They just don't appear in normal computation, because the
interpreter raises an exception instead.
What exception? Apparently your claims about NaN in Python are all
wrong -- have you been using a custom interpreter?
Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type help, copyright, credits or license for more information.
float('inf') - float('inf')
nan
--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, 08 Jul 2014 20:31:25 +0300, Marko Rauhamaa wrote:
Thus, dict operations never test NaN == NaN
You're assuming that there is only one NAN instance. That is not correct:
py a = float('nan')
py b = float('nan')
py a is b
False
py a in {b: None}
False
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, 08 Jul 2014 19:16:53 +0300, Marko Rauhamaa wrote:
Chris Angelico ros...@gmail.com:
Why *should* all NaNs be equal to each other?
I appreciate why you can't say NaN is equal to NaN. However, shouldn't
the very comparison attempt trigger an arithmetic exception?
No. Why should it? It's not an error to check whether two things are
equal.
After all, so does a division by zero.
Um, yes, and multiplying by zero isn't an error. In what way is x == y
related to x/0 ?
But having said that, sometimes it is useful to stop processing as soon
as you reach a NAN. For that, IEEE-754 defines two kinds of NANs, Quiet
NANs and Signalling NANs. Quiet NANs don't trigger a signal when you
perform operations on them. (By default -- I believe you can enable
signals if you wish.) Signalling NANs always trigger a signal, including
when you check them for equality:
py from decimal import Decimal as D
py a = D('nan')
py b = D('snan')
py 1 == a
False
py 1 == b
Traceback (most recent call last):
File stdin, line 1, in module
decimal.InvalidOperation: [class 'decimal.InvalidOperation']
But by default, NANs are quiet. The C99 standard doesn't support
signalling NANs, and Java actually prohibits them.
Aside: The influence of C and Java has crippled IEEE-754 support across
almost all languages and hardware. It's a crying shame the pernicious
influence those two languages have had.
http://grouper.ieee.org/groups/1788/email/pdfmPSi1DgZZf.pdf
Floating point is *hard*, and people who don't understand it insist on
telling those who do that you don't need that feature :-(
--
Steven
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Chris Angelico wrote:
Why *should* all NaNs be equal to each other? You said on the other
list that NaN==NaN was equivalent to (2+2)==(1+3), but that assumes
that NaN is a single thing.
I don't actually care if all NaN bitpatterns are in the same equivalence group
or if each bitpattern is its own equivalence group. I just want the ==
equivalence relation to be sound.
For hash keys, float object identity will successfully look them up:
Except you can't expect to rely on object identity in most interesting cases.
x = float('nan')
import struct
y = struct.unpack('f', struct.pack('f', x))[0]
d[x] = found
d[y]
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: nan
and also:
def f(): return float('inf')/float('inf')
f() == f()
False
f() is f()
False
But any time you compare floats for equality, you *already* have to
understand what you're doing (because of rounding and such), so I
don't see why the special case on NaN is significant, unless as
mentioned above, you want all NaNs to be equal, which doesn't make
sense.
Let me conjure up a simple example:
| class Monitor(Thread):
| def run(self):
| old = self.get_current_value()
| while not self.Terminated:
| new = self.get_current_value()
| if new != old:
| print(time.asctime(), changed to, new)
| old = new
| time.sleep(1)
This is a completely generic change detection algorithm, and not a
floating-point algorithm in any way: It will work on strings, lists, sets,
anything that get_current_value returns, including non-NaN floats. You don't
need to know anything about floating-point representation to write or use such
an algorithm, why should you? It works on tuples, sets, lists, serial port
handles, module objects, pretty much anything you can imagine -- except NaN floats.
regards, Anders
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Wed, Jul 9, 2014 at 4:29 AM, Anders J. Munch 2...@jmunch.dk wrote:
Chris Angelico wrote:
For hash keys, float object identity will successfully look them up:
Except you can't expect to rely on object identity in most interesting
cases.
x = float('nan')
import struct
y = struct.unpack('f', struct.pack('f', x))[0]
d[x] = found
d[y]
Traceback (most recent call last):
File stdin, line 1, in module
KeyError: nan
and also:
def f(): return float('inf')/float('inf')
f() == f()
False
f() is f()
False
Neither of those examples is anything to do with object identity. In
the first case, you pack the value and recreate an equivalent of the
same type with the same value. In the second, you independently
construct two floats with the same value. Both times, you have
equivalence (both are NaN), but not identical objects.
Let me conjure up a simple example:
| class Monitor(Thread):
| def run(self):
| old = self.get_current_value()
| while not self.Terminated:
| new = self.get_current_value()
| if new != old:
| print(time.asctime(), changed to, new)
| old = new
| time.sleep(1)
This is a completely generic change detection algorithm, and not a
floating-point algorithm in any way: It will work on strings, lists, sets,
anything that get_current_value returns, including non-NaN floats. You don't
need to know anything about floating-point representation to write or use
such an algorithm, why should you? It works on tuples, sets, lists, serial
port handles, module objects, pretty much anything you can imagine -- except
NaN floats.
You also have to cope with objects that define __eq__ or __ne__. If
you know that you have a monitor like that, then you make sure that
the value is something comparable; there are plenty of reasons to have
to be careful there. (Consider, for instance, what happens if
get_current_value() returns a list. Mutating that list can change
whether or not it's equal to something else.)
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Ian Kelly wrote: As far as I know nothing changed between 2.4 and 2.7 in this regard. Python has always had NaN compare unequal to everything, per the standard. It might have been platform-specific in 2.4. Okay, here's your problem: there isn't just one binary representation for NaN. I'm fully aware of that. Whether NaN's are one equivalence class or several is not the issue. What matters is the integrity of the equivalence relation. Following the standard isn't a good reason itself? If a standard tells you to jump of a cliff... regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Steven D'Aprano wrote: Oh, you've read the IEEE-754 standard, and that's what it says? We're going to specify this behaviour for NANs just to annoy people perhaps? I was referring to the deliberate choice to enforce IEEE-754 rules in Python. There is no force of law that requires Python to do so. regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Ethan Furman skrev:
What exception? Apparently your claims about NaN in Python are all wrong --
have you been using a custom interpreter?
float('inf') - float('inf')
nan
If you deliberately try to manufacture NaN's, you can. I never said otherwise.
regards, Anders
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Den 14-07-08 19:23, Skip Montanaro skrev: In addition to what others have written, I will add one thing. There are certainly situations where raising an exception is bad. Consider all the people in the scientific computing community doing fancy linear algebra sorts of things, often with missing data. They generally want NaN propagated and not have some long running calculation crash in the middle. NaN!=NaN doesn't cause NaN's to propagate any more or any less. It simply causes a different branch of code to run, quite often the wrong one. regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
I wrote: Steven D'Aprano wrote: Oh, you've read the IEEE-754 standard, and that's what it says? We're going to specify this behaviour for NANs just to annoy people perhaps? I was referring to the deliberate choice to enforce IEEE-754 rules in Python. There is no force of law that requires Python to do so. And just to be clear, I didn't mean python-dev did this to annoy people either. I just meant that the choice made is not supported by any use case, so there's no upside to outweigh the problems it creates. At least I've yet to hear any use case. So far I'm 0 beers in debt. regards, Anders -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 07/08/2014 12:04 PM, Anders J. Munch wrote:
Ethan Furman skrev:
What exception? Apparently your claims about NaN in Python are all wrong --
have you been using a custom interpreter?
float('inf') - float('inf')
nan
If you deliberately try to manufacture NaN's, you can. I never said otherwise.
What you said is: They just don't appear in normal computation, because the
interpreter raises an exception instead.
I just ran a calculation that created a NaN, the same as 4 - 3 creates a 1, and
no exception was raised.
Do you have an example where one is?
--
~Ethan~
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, Jul 8, 2014 at 1:16 PM, Ethan Furman et...@stoneleaf.us wrote: What you said is: They just don't appear in normal computation, because the interpreter raises an exception instead. I just ran a calculation that created a NaN, the same as 4 - 3 creates a 1, and no exception was raised. Do you have an example where one is? The math module functions raise exceptions instead of returning NaN. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, Jul 8, 2014 at 12:54 PM, Anders J. Munch 2...@jmunch.dk wrote: Ian Kelly wrote: As far as I know nothing changed between 2.4 and 2.7 in this regard. Python has always had NaN compare unequal to everything, per the standard. It might have been platform-specific in 2.4. I doubt it, but okay, which platform? Okay, here's your problem: there isn't just one binary representation for NaN. I'm fully aware of that. Whether NaN's are one equivalence class or several is not the issue. What matters is the integrity of the equivalence relation. I have some bad news for you. This is on Python 2.7.6: from decimal import Decimal from fractions import Fraction Decimal(2) == 2 True 2 == Fraction(2) True Decimal(2) == Fraction(2) False I'm not sure exactly when this bug was fixed, but it works as expected in 3.4.0. Following the standard isn't a good reason itself? If a standard tells you to jump of a cliff... So I don't know of a good use case for nan != nan in Python (but really I'm not the one to ask), but I do know of use cases in other scenarios. On platforms that don't provide an isnan() function, the only convenient and efficient way to test for nan is by testing reflexivity: x != x. Following the standard means that any algorithm that uses this trick can (in theory) be implemented in Python without changes. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 07/08/2014 12:50 PM, Ian Kelly wrote: On Tue, Jul 8, 2014 at 1:16 PM, Ethan Furman et...@stoneleaf.us wrote: What you said is: They just don't appear in normal computation, because the interpreter raises an exception instead. I just ran a calculation that created a NaN, the same as 4 - 3 creates a 1, and no exception was raised. Do you have an example where one is? The math module functions raise exceptions instead of returning NaN. Ah, so if actual numbers go in, either actual numbers come out or an exception is raised -- good to know, and thanks for clarifying. -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 07/08/2014 11:54 AM, Anders J. Munch wrote: If a standard tells you to jump of a cliff... because a bunch of tanks are chasing you down, there's water at the bottom, and not a helicopter in sight... well, jumping off the cliff could easily be your best chance. ;) -- ~Ethan~ -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, Jul 8, 2014 at 4:03 PM, Ethan Furman et...@stoneleaf.us wrote: On 07/08/2014 11:54 AM, Anders J. Munch wrote: If a standard tells you to jump of a cliff... because a bunch of tanks are chasing you down, there's water at the bottom, and not a helicopter in sight... well, jumping off the cliff could easily be your best chance. ;) Especially if you have a friend with a TARDIS. -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 08/07/2014 23:07, Ian Kelly wrote: On Tue, Jul 8, 2014 at 4:03 PM, Ethan Furman et...@stoneleaf.us wrote: On 07/08/2014 11:54 AM, Anders J. Munch wrote: If a standard tells you to jump of a cliff... because a bunch of tanks are chasing you down, there's water at the bottom, and not a helicopter in sight... well, jumping off the cliff could easily be your best chance. ;) Especially if you have a friend with a TARDIS. This can lead to unforeseen circumstances though http://en.wikipedia.org/wiki/The_Flipside_of_Dominick_Hide -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, 08 Jul 2014 21:02:09 +0200, Anders J. Munch wrote: Steven D'Aprano wrote: Oh, you've read the IEEE-754 standard, and that's what it says? We're going to specify this behaviour for NANs just to annoy people perhaps? I was referring to the deliberate choice to enforce IEEE-754 rules in Python. There is no force of law that requires Python to do so. There's no force of law that requires Python to enforce reflexivity on values where reflexivity does not apply, any more than Python should enforce total order on values which aren't ordered (such as complex numbers, or sets). I'm sorry that you happened to (apparently) have a use-case where you simultaneously have to deal with NANs but not in a numeric context. But from the description of your problem, it seems to me that the obvious solution is not to deal with floats until as late as possible. That is, your current work-flow (if I have understood it correctly) is: * gather unpacked floats from some device, as ints * pack them into floats * process them in some way which requires reflexivity * (maybe) much later perform numeric calculations on them It seems to me that the trivial work-around is: * gather packed floats from some device, as ints * process them *as ints* in some way which requires reflexivity * unpack back into floats * (maybe) much later perform numeric calculations on them Although perhaps I don't understand your use-case. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, 08 Jul 2014 13:50:06 -0600, Ian Kelly wrote: The math module functions raise exceptions instead of returning NaN. That's not the only way that NANs can be constructed. In particular, the OP is collecting NANs from some other device. I don't know how it ends up with NANs, but it does, so he has to deal with them. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On Tue, 08 Jul 2014 13:57:30 -0600, Ian Kelly wrote: Decimal(2) == Fraction(2) False I'm not sure exactly when this bug was fixed, but it works as expected in 3.4.0. It isn't a bug, it's a missing feature. The problem is that in Python 2.7, neither Decimal nor Fraction are aware of the other: py Decimal(2).__eq__(Fraction(2)) NotImplemented py Fraction(2).__eq__(Decimal(2)) NotImplemented The obvious, but wrong, solution is to coerce both into floats: py a = Decimal(2.01) py b = Fraction(2) - Fraction(1, 2)**100 py a 2 b # a and b are certainly not equal True py float(a) == float(b) True Some time by Python 3.3, Decimal appears to have become aware of how to compare exactly with Fraction. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
[OT] Grandma, is that you? [was Re: NaN comparisons - Call For Anecdotes]
On Wed, 09 Jul 2014 00:21:22 +0100, Mark Lawrence wrote: This can lead to unforeseen circumstances though http://en.wikipedia.org/wiki/The_Flipside_of_Dominick_Hide I prefer this one: http://en.wikipedia.org/wiki/Roswell_That_Ends_Well -- https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
On 7/8/2014 8:10 PM, Steven D'Aprano wrote:
There's no force of law that requires Python to enforce reflexivity on
values where reflexivity does not apply,
There are laws of logic that make the lack of reflexivity obnoxious when
putting objects in collections. Python evaded the problem, at least for
some builtins, by contradicting itself and treating nans as equal to
themselves in the context of collections.
In 2.x, 'in' was defined in terms of ==, but
nan=float('nan')
nl = [nan]
nan in nl
True
even though nan != the only member of nl.
In 3.x, 'in' was redefined to include 'is' as well as '=='.
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Re: NaN comparisons - Call For Anecdotes
Ian Kelly wrote: well, jumping off the cliff could easily be your best chance. ;) Especially if you have a friend with a TARDIS. Or you're James Bond, in which case you ski off the cliff and then open your parachute. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
