[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[Wostr]

Wostr added a comment.

In T95425#2410249, @Nikki wrote:
This just came up on-wiki again at https://www.wikidata.org/wiki/Wikidata:Project_chat#Rounding_when_uncertainty_over_10_is_given where someone added a statement with 547±17 (which is 530-564) and instead it displays 550±20 (which is 530-570).


That's about my comment.
The present situation is unacceptable. It does not matter if the value is stored correctly or not while it is displayed completely wrong and can be misleading. I see no reason for that the automatic mechanism could decide, when and how the values are rounded. That's an absurd. The mechanism does not know when the last digits are significant or not. We can't even precise what type of uncertainty it is.

In this case, the source stated 547±17 for some reason. Not 550±20. On what basis the mechanism round the two significant digits of uncertainty to useless "20"? Your mechanism cannot be smarter than the human. In my field of work such "rounding" may be very annoying, costly or even dangerous, becasue the "insignificant digits" matters.TASK DETAILhttps://phabricator.wikimedia.org/T95425EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: WostrCc: Wostr, Srittau, Nikki, Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, Aklapper, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[Nikki]

Nikki added a comment.
This just came up on-wiki again at https://www.wikidata.org/wiki/Wikidata:Project_chat#Rounding_when_uncertainty_over_10_is_given where someone added a statement with 547±17 (which is 530-564) and instead it displays 550±20 (which is 530-570).TASK DETAILhttps://phabricator.wikimedia.org/T95425EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: NikkiCc: Srittau, Nikki, Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, Aklapper, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[thiemowmde]

thiemowmde added a comment.

F3239746: Tanker_in_English_Channel_2.png 



TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: thiemowmde
Cc: Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, 
thiemowmde, Aklapper, Wikidata-bugs, aude, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[thiemowmde]

thiemowmde added a comment.

This argument is invalid for multiple reasons:

1. It does not matter if you call it a "range" or a "possibility". For all 
arguments shared that's the same.
  - In 1.5±1.0 it's possible that the tanker is sitting somewhere between 0.5 
and 2.5. But it's **impossible** to sit at 3.0.
  - In 2±1 all possibilities do a magic jump of 0.5 to the right, rendering 0.5 
impossible and 3.0 possible. This is just wrong; read: this is neither what 
1.5±1.0 means nor what the user meant when he entered this. As I tried to 
illustrate above an added error of 0.5 is highly significant. This is 
equivalent to 33% of the value or 50% of the precision.
2. What we call "precision" is not an integer. It's a floating point number. It 
does not say anything about "digits" and how "significant" they are. I think I 
already repeated this more often than I wanted to: there is no such thing as an 
"irrelevant" digit in here. 1.25+/-0.5 does not mean something like "half of 
the first digit after the decimal point is irrelevant". What is this even 
supposed to mean? How would you display this, so a user can read and understand 
it? Oh, I know: What about 1.25+/-0.5? What's unclear about this? Why do values 
with precision need rounding at all?

I propose this as an acceptance criteria for an acceptable formatter: Silent 
manipulations of more than 1% of the original value are unacceptable.

In general: Situations where code thinks it knows better than the user are 
unacceptable.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: thiemowmde
Cc: Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, 
thiemowmde, Aklapper, Wikidata-bugs, aude, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[daniel]

daniel added a comment.

@thiemowmde: That's what happens when you use precision to express a range. I 
agree that it's confusing. +/-1 says that anything after the decimal point is 
insignificant. If you construct a range from that, the result is 
counter-intuitive.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, 
thiemowmde, Aklapper, Wikidata-bugs, aude, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[thiemowmde]

thiemowmde added a comment.

F3230294: Tanker_in_English_Channel.png 



TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: thiemowmde
Cc: Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, 
thiemowmde, Aklapper, Wikidata-bugs, aude, Mbch331



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[thiemowmde]

thiemowmde added a comment.

@daniel wrote:

> The reason 350±150 is shown as  400±200 is that rounding is applied based on 
> the uncertainty


I'm afraid this does not explain anything. The uncertainty is +/-150. Not 
+/-200. Rounding to +/-150 would mean ... rounding to 2 * 150 = 300? But why? 
The value is 350. What's wrong with stating that 350+/-150 is 350+/-150? What 
do we win by stating that 350+/-150 is 400+/-200?

And why round the uncertainty based on ... what? Based on the uncertainty? How 
does this make sense?

Converting 350+/-150 feet to meter results in 106.68+/-45.72 meter. There is 
neither anything wrong with 350+/-150 feet nor with 106.68+/-45.72 meter. We 
can think about suppressing irrelevant digits and render this as a value that 
survives a roundtrip back to the original unit, e.g. 107+/-46 meter. Or 
rounding to 1% of the uncertainty. But this is not what happens currently.

Sure, an output like "106.68 meter" without the uncertainty would indeed 
introduce "false precision". But even this does not mean we must round this to 
100. The error introduced by this is worse than the error by the false 
precision. We must still round this converted value to something that survives 
a roundtrip back to the original unit with an acceptable error of about 1%.

> we have never [...] tried to make the HTML representation lossless.


I'm not sure where this comes from. All string types are rendered in a 
perfectly lossless format. Identifiers, URLs, Commons media, all lossless. Time 
values are usually lossless, with rare edge cases.

> references to other entities [...] when you copy&paste them, you lose 
> information.


You copy them by right click and copying the id in the URL.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: thiemowmde
Cc: Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, 
thiemowmde, Aklapper, Wikidata-bugs, aude



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[daniel]

daniel added a comment.

The reason 350±150 is shown as  400±200 is that rounding is applied based on 
the uncertainty, and the same rounding is applied to the uncertainty itself 
(basically, you cannot be more precise about the uncertainty than about the 
value itself). The reason we apply rounding here is for consistency: if unit 
conversion is applied, we have to apply rounding to avoid false precision. So 
for consistency, we always apply rounding.

But this is not set in stone. I'm coming around to the opinion to apply 
rounding only when needed, and to show values as-is otherwise.

Regarding the "data loss" aspect of the HTML rendering I disagree: we have 
never guaranteed or even tried to make the HTML representation lossless. That 
would be rather hard to do, and would contradict the wish to have "nice" 
readable values in HTML. This is particularly true for values that are 
references to other entities - we show them using labels, not IDs, so when you 
copy&paste them, you lose information.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, 
thiemowmde, Aklapper, Wikidata-bugs, aude



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[Thryduulf]

Thryduulf added a comment.

In https://phabricator.wikimedia.org/T95425#1624288, @daniel wrote:

> As far as I know, this is resolved for the editing use case. Rounding still 
> applies for HTML output. I think this should be either reworded or closed.


This is still causing data loss, just for those who use the shown data rather 
than those who use the API stored value so the title still seems correct to me.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Thryduulf
Cc: Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, 
thiemowmde, Aklapper, Wikidata-bugs, aude



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[Thryduulf]

Thryduulf added a subscriber: Thryduulf.
Thryduulf added a comment.

This is still causing incorrect data to be displayed.
I've entered a value of 350±150 because the source gives a range of 200-500. 
However this is displayed as 400±200 which gives a range of 200-600 which is 
incorrect and misleading.

I don't see why there is any need or justification for displaying the output 
differently to the stored value - even more so when the difference is so 
significant (why is there ever need to round to the nearest 100 for values of 
this magnitude? The nearest 10 would be understandable (but still wrong) but 
removing an order of magnitude more precision than that is baffling).


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Thryduulf
Cc: Thryduulf, Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, 
thiemowmde, Aklapper, Wikidata-bugs, aude



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[daniel]

daniel added a comment.

As far as I know, this is resolved for the editing use case. Rounding still 
applies for HTML output. I think this should be either reworded or closed.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, 
Aklapper, Wikidata-bugs, aude



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[Jc3s5h]

Jc3s5h added a comment.

In https://phabricator.wikimedia.org/T95425#1606173, @thiemowmde wrote in part:

> Proof of concept: https://github.com/DataValues/Number/pull/43


I don't fully understand this example, but I did notice all the examples seem 
to provide displayed uncertainties that are an exact power of 10. I would think 
that if //amount = 25, lowerBound = 23, upperBound = 27// were stored in the 
database then it should be rendered as 25 ± 2.

> > principle of consistency

> 

> 

> Wut? How is the example in https://phabricator.wikimedia.org/T95425#1601799 
> "consistent"?


I'm not terribly familiar with the user interface of phabricator. I don't know 
how to follow the link https://phabricator.wikimedia.org/T95425#1601799 to get 
to the exact spot you are referring to.

> @Jc3s5h wrote:

> 

> > As an example of what's wrong with time [...] The death date of Benjamin 
> > Franklin [...]

> 

> 

> Hurray, an other discussion about time issues in a ticket where it does not 
> belong, where it won't be read, can not be considered and does not do 
> anything but waste developers time when they have to figure out what the 
> relevance for the issue in this ticket is. Hint: It's zero.


You have missed my point, which is that since time is so messed up we should 
totally disregard how time represents uncertainty while working on representing 
the uncertainty of quantities.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, 
Aklapper, Wikidata-bugs, aude



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[thiemowmde]

thiemowmde added a comment.

> for the same reason 1964 with the precision set to century will be rendered 
> as 20th century, not 1964+/-50.


Irrelevant, misleading comparison. Quantities do not have a precision, they 
have a lower and upper bound.

> principle of consistency


Wut? How is the example in https://phabricator.wikimedia.org/T95425#1601799 
"consistent"?

> not be a decision on the technical level.


So you refuse to call the broken behavior I showed in 
https://phabricator.wikimedia.org/T95425#1601799 a bug? Again, I don't know how 
this could be more obvious.

@Jc3s5h wrote:

> As an example of what's wrong with time [...] The death date of Benjamin 
> Franklin [...]


Hurray, an other discussion about time issues in a ticket where it does not 
belong, where it won't be read, can not be considered and does not do anything 
but waste developers time when they have to figure out what the relevance for 
the issue in this ticket is. Hint: It's zero.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: thiemowmde
Cc: Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, 
Aklapper, Wikidata-bugs, aude



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[daniel]

daniel added a comment.

@Jc3s5h I don't see how any of the differences you mention are relevant in the 
context of precision/uncertainty.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, 
Aklapper, Wikidata-bugs, aude, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[Jc3s5h]

Jc3s5h added a comment.

In https://phabricator.wikimedia.org/T95425#1602763, @daniel wrote in part:

> @thiemowmde for the same reason 1964 with the precision set to century will 
> be rendered as 20th century, not 1964+/-50.


If you want a solution to be analogous to the way time is handled, then you 
need to fix time first. I think it would be a mistake to even think about time 
for this purpose because it's going to take time to fix time.

As an example of what's wrong with time, the user interface does not allow 
entering time zone. Apparently, the bots that add birth and death dates took 
their cue from the user interface, and didn't add it. So every birth or death 
date with a precision of 1 day is wrong, except for those people who were born 
when and where the time zone offset was 0, such as the United Kingdom in winter.

Time and other measurements are different from each other in the way people 
think about them. For example, if I have a credit card that's 85 mm long from 
Hartford CT to London, it's still 85 mm long. If I go to a bar in New Zealand 
at 2 pm Sept 4, and show my passport showing I was born in Connecticut USA on 
Sept 4, 1997, the bartender will serve me even though it isn't September 4 yet 
in Connecticut. So I think we are going to need different approaches for time 
and other quantities.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, 
Aklapper, Wikidata-bugs, aude, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[daniel]

daniel added a comment.

@thiemowmde for the same reason 1964 with the precision set to century will be 
rendered as 20th century, not 1964+/-50.

Basically, we have two conflicting principles here: the principle of least 
surprise agrees with you thiemo (I entered one thing, but see another, wtf?), 
and the principle of consistency (wrt rounding) points to my conclusion. In the 
end, this is a product level decision. The current approach was decided by 
Denny a long time ago. We can change it, but that would not be a decision on 
the technical level.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: Addshore, Lydia_Pintscher, Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, 
Aklapper, Wikidata-bugs, aude, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[Jc3s5h]

Jc3s5h added a comment.

In https://phabricator.wikimedia.org/T95425#1602377, @daniel wrote in part:

>




> We could just drop rounding, but that would lead to //false precision// when 
> applying unit conversion. So we could apply rounding only if we do conversion 
> - but that would be even more confusing, don't you think? And of course, if 
> we do conversion, round trips will never work.


Whether rounding only on conversion would be confusing depends on the 
background of the reader. The casual reader will be confused by many 
significant digits (or apparently significant digits) no matter what we do. The 
reader with a rigorous quantitative background such as scientists and engineers 
will expect the original value would have been entered correctly, and will 
recognize the inherent data loss involved in conversion. I think such a reader 
would expect us to preserve precision when some cases will require such 
preservation to properly present the original value (even if badly entered 
values will look bad). Such a user will expect us to round on conversion 
because that is customary among people with a strong quantitative background.

Preserving the input when there is no conversion also helps to bring 
badly-entered original values to the attention of editors who can then fix them.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Jc3s5h
Cc: Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, Aklapper, Wikidata-bugs, 
aude, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T95425: [Bug] Quantity formatter rounding causes significant data loss

[daniel]

daniel added a comment.

Round trip stability for rendered output is nice, but was never a design goal. 
In fact, it doesn't work for most things. 
If this is not about editing, saying it causes "data loss" is misleading. 
"Quantity rendering does not preserve precision" would be descriptive. But 
that'S not a bug, that's working as designed. Rounding is lossy per definition.

We could just drop rounding, but that would lead to //false precision// when 
applying unit conversion. So we could apply rounding only if we do conversion - 
but that would be even more confusing, don't you think? And of course, if we do 
conversion, round trips will never work.


TASK DETAIL
  https://phabricator.wikimedia.org/T95425

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: daniel
Cc: Jc3s5h, Snipre, mgrabovsky, daniel, thiemowmde, Aklapper, Wikidata-bugs, 
aude, Malyacko



___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs