Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi all, Sorry, I was a bit busy during August. On Wed, Aug 5, 2015 at 5:37 PM, Yasuo Ohgakiwrote: > I sent work in progress PR for this and updated the RFC. > > https://github.com/php/php-src/pull/1455 > TODO: Add/modify tests. Add 0 mode support for PG(precision). Add > WDDX/XMLRPC PG(serialize_precision) support. > > https://wiki.php.net/rfc/precise_float_value The patch and wiki is updated. It's better to minimize the change at this time. I didn't look into WDDX/XMLRPC which uses EG(precision), if anyone insist, I'll look into it. I don't mind much if anyone suggests to drop PHP 5.6 part. I'll drop the part before RFC discussion to focus PHP7. It should only affects php_gcvt() and php.ini's precision setting. i.e. var_dump/var_export/ json_encode/echo/print. Tests passes, but if anyone notice misbehavior, please let me know. I would like to start RFC discussion in a few days. Thank you. -- Yasuo Ohgaki yohg...@ohgaki.net -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi all, On Fri, Jul 31, 2015 at 4:44 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: On Thu, Jul 30, 2015 at 6:06 PM, Nikita Popov nikita@gmail.com wrote: On Thu, Jul 30, 2015 at 1:25 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi all, On Thu, Jul 30, 2015 at 7:44 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita@gmail.com wrote: Instead of continuing to use serialize_precision, which will produce unnecessarily long outputs for many values, why don't we just switch to using the 0 mode of zend_dtoa, i.e. to return the shortest output that is still accurate if interpreted in round-to-nearest. I think this is what everybody else is using when they convert floating point numbers to strings. I guess we may not be able to change normal floating point printing to use this, but this seems like the best mode for anything using serialize_precision now and everything that should be using it (like JSON, and queries, etc). I prefer your proposal! Your proposal is a lot better than now. Anyone has opinion for this? I'm writing the RFC and I would like to make this the first option. i.e. serialize_precision=0 uses zend_dtoa 0 mode for all data exchange functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?) I wrote draft RFC. https://wiki.php.net/rfc/precise_float_value Please comment. I would like to start RFC discussion shortly. Thank you. Nice idea about using a special serialize_precision value for this. This allows to keep BC for those that have tests for particular serialize output or similar things. I would suggest to default serialize_precision to -1 in PHP 7 -- if people want the previous behavior they can still have it, but I think -1 is the more reasonable default as it matches what one would naturally expect. I don't see the need for having a separate setting for JSON. Having a dozen different float precision settings will not help anyone. Thank you for your feedback. I agree it's better if PHP7 uses 0 mode by default. I'll update the RFC. I have things to do on this weekend. I'll try to write patch this weekend hopefully. I sent work in progress PR for this and updated the RFC. https://github.com/php/php-src/pull/1455 TODO: Add/modify tests. Add 0 mode support for PG(precision). Add WDDX/XMLRPC PG(serialize_precision) support. https://wiki.php.net/rfc/precise_float_value Old behavior MacBook-Pro:github-php-src yohgaki$ php -r '$a=[v=123450.1234567890123]; var_dump(json_encode($a), serialize($a), var_export($a, true));'; string(16) {v:1.2345e+29} string(39) a:1:{s:1:v;d:1.2344E+29;} string(42) array ( 'v' = 1.2344E+29, ) New behavior MacBook-Pro:github-php-src yohgaki$ ./php-bin -r '$a=[v=123450.1234567890123]; var_dump(json_encode($a), serialize($a), var_export($a, true));'; string(16) {v:1.2345e+29} string(27) a:1:{s:1:v;d:1.2345E+29;} string(30) array ( 'v' = 1.2345E+29, ) Before I start working on PG(precision) 0 mode support, I would like to hear comments for this change if any. The default (PG(precision)=14) will not be changed and there will be no BC. I think it's more consistent if both PG(precision) and PG(serialize_precision) support 0 mode. Any comments for adding PG(precision) 0 mode support? Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Nikita, On Thu, Jul 30, 2015 at 6:06 PM, Nikita Popov nikita@gmail.com wrote: On Thu, Jul 30, 2015 at 1:25 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi all, On Thu, Jul 30, 2015 at 7:44 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita@gmail.com wrote: Instead of continuing to use serialize_precision, which will produce unnecessarily long outputs for many values, why don't we just switch to using the 0 mode of zend_dtoa, i.e. to return the shortest output that is still accurate if interpreted in round-to-nearest. I think this is what everybody else is using when they convert floating point numbers to strings. I guess we may not be able to change normal floating point printing to use this, but this seems like the best mode for anything using serialize_precision now and everything that should be using it (like JSON, and queries, etc). I prefer your proposal! Your proposal is a lot better than now. Anyone has opinion for this? I'm writing the RFC and I would like to make this the first option. i.e. serialize_precision=0 uses zend_dtoa 0 mode for all data exchange functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?) I wrote draft RFC. https://wiki.php.net/rfc/precise_float_value Please comment. I would like to start RFC discussion shortly. Thank you. Nice idea about using a special serialize_precision value for this. This allows to keep BC for those that have tests for particular serialize output or similar things. I would suggest to default serialize_precision to -1 in PHP 7 -- if people want the previous behavior they can still have it, but I think -1 is the more reasonable default as it matches what one would naturally expect. I don't see the need for having a separate setting for JSON. Having a dozen different float precision settings will not help anyone. Thank you for your feedback. I agree it's better if PHP7 uses 0 mode by default. I'll update the RFC. I have things to do on this weekend. I'll try to write patch this weekend hopefully. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
On Thu, Jul 30, 2015 at 1:25 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi all, On Thu, Jul 30, 2015 at 7:44 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita@gmail.com wrote: Instead of continuing to use serialize_precision, which will produce unnecessarily long outputs for many values, why don't we just switch to using the 0 mode of zend_dtoa, i.e. to return the shortest output that is still accurate if interpreted in round-to-nearest. I think this is what everybody else is using when they convert floating point numbers to strings. I guess we may not be able to change normal floating point printing to use this, but this seems like the best mode for anything using serialize_precision now and everything that should be using it (like JSON, and queries, etc). I prefer your proposal! Your proposal is a lot better than now. Anyone has opinion for this? I'm writing the RFC and I would like to make this the first option. i.e. serialize_precision=0 uses zend_dtoa 0 mode for all data exchange functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?) I wrote draft RFC. https://wiki.php.net/rfc/precise_float_value Please comment. I would like to start RFC discussion shortly. Thank you. Nice idea about using a special serialize_precision value for this. This allows to keep BC for those that have tests for particular serialize output or similar things. I would suggest to default serialize_precision to -1 in PHP 7 -- if people want the previous behavior they can still have it, but I think -1 is the more reasonable default as it matches what one would naturally expect. I don't see the need for having a separate setting for JSON. Having a dozen different float precision settings will not help anyone. Nikita
Re: [PHP-DEV] json_decode/encode should return full precision values by default
On Wed, Jul 29, 2015 at 11:09 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi Jakub, For me, JSON is one of a data exchange format just like serialize/var_export. Anyway, we are about to reach an agreement. On Wed, Jul 29, 2015 at 5:32 PM, Jakub Zelenka bu...@php.net wrote: Question is Is this the way it should be?. I have already said that using precision ini wasn't the best idea. However json_encode is not the same as serialize and we should not ever change its output in a bug fixing release. Doing that could cause also other issues as this is a BC break. Lets imagine that someone set low precision on purpose just to limit precision and save some space I've been fixed var_export precision issue as a bug. [yohgaki@dev php-src]$ git show 3cf2682083fc1c8635b02c4c commit 3cf2682083fc1c8635b02c4cf77bdf12c5e5da35 Merge: 5c89d5a 4c45e95 Author: Yasuo Ohgaki yohg...@php.net Date: Tue Oct 29 17:30:58 2013 +0900 Merge branch 'PHP-5.5' * PHP-5.5: Fixed Bug 64760 var_export() does not use full precision for floating-point numbers Again it's not the same. json_encode is much more used the var_export IMHO so the chance that you break someone's code is much bigger. Also it doesn't mean that you did the right thing and it's not a precedence for such change in json. I guess that there wasn't anyone who would disagree with you at that time. I agree that JSON is used more than var_export or even serialize. I guess almost all users do not care much about preciseness of JSON numeric. I was the one also until I had to deal with float values. Since almost nobody cares about preciseness, why not make it more precise? I may ask general list see if there are users who mind this change. Although I don't strongly insist merging fix to 5.6 branch, I think data exchange function that is not trying to be precise is bad thing. when transferring data . If you change it, then it's screwed up because it will use different ini. We don't know what people do in their code and we should not break it. As I said this is not a bug but we could consider changing that if the RFC proposing such change passes. Strong json numeric validator may have problem with the change. However, JSON RFC states 6. Numbers This specification allows implementations to set limits on the range and precision of numbers accepted. Since software that implements IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these provide, in the sense that implementations will approximate JSON numbers within the expected precision. A JSON number such as 1E400 or 3.141592653589793238462643383279 may indicate potential interoperability problems, since it suggests that the software that created it expects receiving software to have greater capabilities for numeric magnitude and precision than is widely available. Note that when such software is used, numbers that are integers and are in the range [-(2**53)+1, (2**53)-1] are interoperable in the sense that implementations will agree exactly on their numeric values. https://tools.ietf.org/html/rfc7159 In order PHP to be more compatible, we should use IEEE 754 range or our other data exchange standard, which is serialize_precision=17. We are still compatible. This is recommendation, not requirement. However I have to agree that it will be probably good idea to change default. But just only if agreed in RFC as this is not a bug! Personally I am against changing that to serialize_precision. I'd much prefer introducing new ini (e.g. json_precision) with the same default as serialize_precision. It would give more flexibility and allow changing value just for json. I agree that it is implementation matter how JSON number is treated. Since PHP is general programming language, I think it's better to make it as generic as possible by default. It's perfectly OK for me to have json_precision. I prefer to have it indeed! My position is - PHP7.0: We must use serialize_precision for JSON - PHP5.6: I strongly suggest to use serialize_precision for JSON I'd prefer 7.1 as this is not something that would be so urgent. I guess your reason to postpone the change is json_precision. This could be done by 7.0 as the resolution requires a trivial change to code. I don't see reasons not to fix this issue for 5.6 if serialize_precision is used. As you see we don't agree here. It means that if you want to get it in, please write RFC. You can add the version option as I did it for json numeric to string RFC and you will see if others agree with you that this is a bug. No problem. I'll start RFC discussion shortly. BTW,
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Nikita, On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita@gmail.com wrote: Instead of continuing to use serialize_precision, which will produce unnecessarily long outputs for many values, why don't we just switch to using the 0 mode of zend_dtoa, i.e. to return the shortest output that is still accurate if interpreted in round-to-nearest. I think this is what everybody else is using when they convert floating point numbers to strings. I guess we may not be able to change normal floating point printing to use this, but this seems like the best mode for anything using serialize_precision now and everything that should be using it (like JSON, and queries, etc). I prefer your proposal! Your proposal is a lot better than now. Anyone has opinion for this? I'm writing the RFC and I would like to make this the first option. i.e. serialize_precision=0 uses zend_dtoa 0 mode for all data exchange functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?) Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi all, On Thu, Jul 30, 2015 at 7:44 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita@gmail.com wrote: Instead of continuing to use serialize_precision, which will produce unnecessarily long outputs for many values, why don't we just switch to using the 0 mode of zend_dtoa, i.e. to return the shortest output that is still accurate if interpreted in round-to-nearest. I think this is what everybody else is using when they convert floating point numbers to strings. I guess we may not be able to change normal floating point printing to use this, but this seems like the best mode for anything using serialize_precision now and everything that should be using it (like JSON, and queries, etc). I prefer your proposal! Your proposal is a lot better than now. Anyone has opinion for this? I'm writing the RFC and I would like to make this the first option. i.e. serialize_precision=0 uses zend_dtoa 0 mode for all data exchange functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?) I wrote draft RFC. https://wiki.php.net/rfc/precise_float_value Please comment. I would like to start RFC discussion shortly. Thank you. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
On Wed, Jul 29, 2015 at 5:25 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi all, On Thu, Jul 30, 2015 at 7:44 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: On Thu, Jul 30, 2015 at 1:13 AM, Nikita Popov nikita@gmail.com wrote: Instead of continuing to use serialize_precision, which will produce unnecessarily long outputs for many values, why don't we just switch to using the 0 mode of zend_dtoa, i.e. to return the shortest output that is still accurate if interpreted in round-to-nearest. I think this is what everybody else is using when they convert floating point numbers to strings. I guess we may not be able to change normal floating point printing to use this, but this seems like the best mode for anything using serialize_precision now and everything that should be using it (like JSON, and queries, etc). I prefer your proposal! Your proposal is a lot better than now. Anyone has opinion for this? I'm writing the RFC and I would like to make this the first option. i.e. serialize_precision=0 uses zend_dtoa 0 mode for all data exchange functions (json/serialize/var_exrport. Anyone care about WDDX/XML_RPC?) I wrote draft RFC. https://wiki.php.net/rfc/precise_float_value Please comment. I would like to start RFC discussion shortly. Thank you. My thought would be why not -1 to make it obvious it has special meaning? When I see precision=0 I tend to think that means my numbers will be returned as ints no matter what (there is zero fractional precision) whereas this takes a valid value for precision and gives it special meaning. I fully support your end game, but giving a valid value a special meaning I do not like. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Yasuo, On Tue, Jul 28, 2015 at 10:51 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi Jakub, On Wed, Jul 29, 2015 at 3:15 AM, Jakub Zelenka bu...@php.net wrote: On Mon, Jul 27, 2015 at 11:17 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Get JSON data from Google maps and store the data using PHP, then users lose last 2 digits of fraction part by default. The value is changed and wrong. This is definitely a bug. I don't really get why you use Google maps as an example. Do you actually know what's the difference between two distances that differs after rounding with precision 14? I just tried it on http://www.ig.utexas.edu/outreach/googleearth/latlong.html and set Lat 1: 51.602097123457 ; Long 1: -0.120667 Lat 2: 51.602097123458 ; Long 2: -0.120667 The difference is 1.114e-10 km which is 0.0001114 millimetres. Are you actually serious about that? :) It's error margin for GPS. How about scientific values? This is certainly not a margin error for GPS and definitely has no influence on Google maps... :) Of course there are cases when you can need slightly bigger precision. What I was trying to say is that you exaggerate the problem. I'm not comfortable with that values aren't processed/stored correctly (as much as possible. You know float can't be exactly precise) Even if GPS values may ignore some fractions, I'm very uncomfortable with ignoring/destroying original values while it could be stored exactly as it is. Once again, you have got the option to change it if you don't like the current default. In addition, it's inconsistent with other data exchange functions. i.e. serialize/var_export. json_encode is not serialize nor var_export. It's for different things... I have already said that you might care about output size and if you do, you certainly don't want to be changed in the bug fixing release. Question is Is this the way it should be?. I have already said that using precision ini wasn't the best idea. However json_encode is not the same as serialize and we should not ever change its output in a bug fixing release. Doing that could cause also other issues as this is a BC break. Lets imagine that someone set low precision on purpose just to limit precision and save some space I've been fixed var_export precision issue as a bug. [yohgaki@dev php-src]$ git show 3cf2682083fc1c8635b02c4c commit 3cf2682083fc1c8635b02c4cf77bdf12c5e5da35 Merge: 5c89d5a 4c45e95 Author: Yasuo Ohgaki yohg...@php.net Date: Tue Oct 29 17:30:58 2013 +0900 Merge branch 'PHP-5.5' * PHP-5.5: Fixed Bug 64760 var_export() does not use full precision for floating-point numbers Again it's not the same. json_encode is much more used the var_export IMHO so the chance that you break someone's code is much bigger. Also it doesn't mean that you did the right thing and it's not a precedence for such change in json. I guess that there wasn't anyone who would disagree with you at that time. Although I don't strongly insist merging fix to 5.6 branch, I think data exchange function that is not trying to be precise is bad thing. when transferring data . If you change it, then it's screwed up because it will use different ini. We don't know what people do in their code and we should not break it. As I said this is not a bug but we could consider changing that if the RFC proposing such change passes. Strong json numeric validator may have problem with the change. However, JSON RFC states 6. Numbers This specification allows implementations to set limits on the range and precision of numbers accepted. Since software that implements IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these provide, in the sense that implementations will approximate JSON numbers within the expected precision. A JSON number such as 1E400 or 3.141592653589793238462643383279 may indicate potential interoperability problems, since it suggests that the software that created it expects receiving software to have greater capabilities for numeric magnitude and precision than is widely available. Note that when such software is used, numbers that are integers and are in the range [-(2**53)+1, (2**53)-1] are interoperable in the sense that implementations will agree exactly on their numeric values. https://tools.ietf.org/html/rfc7159 In order PHP to be more compatible, we should use IEEE 754 range or our other data exchange standard, which is serialize_precision=17. We are still compatible. This is recommendation, not requirement. However I have to agree that it will be probably good idea to change default. But just only if agreed in RFC as this is not a bug! Personally I am against changing that to serialize_precision. I'd much prefer introducing new
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Jakub, For me, JSON is one of a data exchange format just like serialize/var_export. Anyway, we are about to reach an agreement. On Wed, Jul 29, 2015 at 5:32 PM, Jakub Zelenka bu...@php.net wrote: Question is Is this the way it should be?. I have already said that using precision ini wasn't the best idea. However json_encode is not the same as serialize and we should not ever change its output in a bug fixing release. Doing that could cause also other issues as this is a BC break. Lets imagine that someone set low precision on purpose just to limit precision and save some space I've been fixed var_export precision issue as a bug. [yohgaki@dev php-src]$ git show 3cf2682083fc1c8635b02c4c commit 3cf2682083fc1c8635b02c4cf77bdf12c5e5da35 Merge: 5c89d5a 4c45e95 Author: Yasuo Ohgaki yohg...@php.net Date: Tue Oct 29 17:30:58 2013 +0900 Merge branch 'PHP-5.5' * PHP-5.5: Fixed Bug 64760 var_export() does not use full precision for floating-point numbers Again it's not the same. json_encode is much more used the var_export IMHO so the chance that you break someone's code is much bigger. Also it doesn't mean that you did the right thing and it's not a precedence for such change in json. I guess that there wasn't anyone who would disagree with you at that time. I agree that JSON is used more than var_export or even serialize. I guess almost all users do not care much about preciseness of JSON numeric. I was the one also until I had to deal with float values. Since almost nobody cares about preciseness, why not make it more precise? I may ask general list see if there are users who mind this change. Although I don't strongly insist merging fix to 5.6 branch, I think data exchange function that is not trying to be precise is bad thing. when transferring data . If you change it, then it's screwed up because it will use different ini. We don't know what people do in their code and we should not break it. As I said this is not a bug but we could consider changing that if the RFC proposing such change passes. Strong json numeric validator may have problem with the change. However, JSON RFC states 6. Numbers This specification allows implementations to set limits on the range and precision of numbers accepted. Since software that implements IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these provide, in the sense that implementations will approximate JSON numbers within the expected precision. A JSON number such as 1E400 or 3.141592653589793238462643383279 may indicate potential interoperability problems, since it suggests that the software that created it expects receiving software to have greater capabilities for numeric magnitude and precision than is widely available. Note that when such software is used, numbers that are integers and are in the range [-(2**53)+1, (2**53)-1] are interoperable in the sense that implementations will agree exactly on their numeric values. https://tools.ietf.org/html/rfc7159 In order PHP to be more compatible, we should use IEEE 754 range or our other data exchange standard, which is serialize_precision=17. We are still compatible. This is recommendation, not requirement. However I have to agree that it will be probably good idea to change default. But just only if agreed in RFC as this is not a bug! Personally I am against changing that to serialize_precision. I'd much prefer introducing new ini (e.g. json_precision) with the same default as serialize_precision. It would give more flexibility and allow changing value just for json. I agree that it is implementation matter how JSON number is treated. Since PHP is general programming language, I think it's better to make it as generic as possible by default. It's perfectly OK for me to have json_precision. I prefer to have it indeed! My position is - PHP7.0: We must use serialize_precision for JSON - PHP5.6: I strongly suggest to use serialize_precision for JSON I'd prefer 7.1 as this is not something that would be so urgent. I guess your reason to postpone the change is json_precision. This could be done by 7.0 as the resolution requires a trivial change to code. I don't see reasons not to fix this issue for 5.6 if serialize_precision is used. As you see we don't agree here. It means that if you want to get it in, please write RFC. You can add the version option as I did it for json numeric to string RFC and you will see if others agree with you that this is a bug. No problem. I'll start RFC discussion shortly. BTW, please don't care if this is a bug or not. We have lots of issues to be resolved. Let's concentrate resolving issues. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Ryan, On Thu, Jul 30, 2015 at 8:35 AM, Ryan Pallas derokor...@gmail.com wrote: I wrote draft RFC. https://wiki.php.net/rfc/precise_float_value Please comment. I would like to start RFC discussion shortly. Thank you. My thought would be why not -1 to make it obvious it has special meaning? When I see precision=0 I tend to think that means my numbers will be returned as ints no matter what (there is zero fractional precision) whereas this takes a valid value for precision and gives it special meaning. I fully support your end game, but giving a valid value a special meaning I do not like. Thank you for your feedback. I'll update the RFC and use -1 for it. When I make the patch for this, I'll post new RFC discussion mail. It will take a while. Comments are appreciated. Regards. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Yasuo On Mon, Jul 27, 2015 at 11:17 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Get JSON data from Google maps and store the data using PHP, then users lose last 2 digits of fraction part by default. The value is changed and wrong. This is definitely a bug. I don't really get why you use Google maps as an example. Do you actually know what's the difference between two distances that differs after rounding with precision 14? I just tried it on http://www.ig.utexas.edu/outreach/googleearth/latlong.html and set Lat 1: 51.602097123457 ; Long 1: -0.120667 Lat 2: 51.602097123458 ; Long 2: -0.120667 The difference is 1.114e-10 km which is 0.0001114 millimetres. Are you actually serious about that? :) We can write $old = ini_set('precision', 17); json_encode($var); ini_set('precision', $old); You can set it once in your ini file if you need such precision. everywhere to workaround this problem. Question is Is this the way it should be?. I have already said that using precision ini wasn't the best idea. However json_encode is not the same as serialize and we should not ever change its output in a bug fixing release. Doing that could cause also other issues as this is a BC break. Lets imagine that someone set low precision on purpose just to limit precision and save some space when transferring data . If you change it, then it's screwed up because it will use different ini. We don't know what people do in their code and we should not break it. As I said this is not a bug but we could consider changing that if the RFC proposing such change passes. Cheers Jakub
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Jakub, On Wed, Jul 29, 2015 at 3:15 AM, Jakub Zelenka bu...@php.net wrote: On Mon, Jul 27, 2015 at 11:17 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Get JSON data from Google maps and store the data using PHP, then users lose last 2 digits of fraction part by default. The value is changed and wrong. This is definitely a bug. I don't really get why you use Google maps as an example. Do you actually know what's the difference between two distances that differs after rounding with precision 14? I just tried it on http://www.ig.utexas.edu/outreach/googleearth/latlong.html and set Lat 1: 51.602097123457 ; Long 1: -0.120667 Lat 2: 51.602097123458 ; Long 2: -0.120667 The difference is 1.114e-10 km which is 0.0001114 millimetres. Are you actually serious about that? :) It's error margin for GPS. How about scientific values? I'm not comfortable with that values aren't processed/stored correctly (as much as possible. You know float can't be exactly precise) Even if GPS values may ignore some fractions, I'm very uncomfortable with ignoring/destroying original values while it could be stored exactly as it is. In addition, it's inconsistent with other data exchange functions. i.e. serialize/var_export. We can write $old = ini_set('precision', 17); json_encode($var); ini_set('precision', $old); You can set it once in your ini file if you need such precision. It's not a good idea. Most PHP codes depend on 14 precision and programmer cannot assure other code works as expected. everywhere to workaround this problem. Question is Is this the way it should be?. I have already said that using precision ini wasn't the best idea. However json_encode is not the same as serialize and we should not ever change its output in a bug fixing release. Doing that could cause also other issues as this is a BC break. Lets imagine that someone set low precision on purpose just to limit precision and save some space I've been fixed var_export precision issue as a bug. [yohgaki@dev php-src]$ git show 3cf2682083fc1c8635b02c4c commit 3cf2682083fc1c8635b02c4cf77bdf12c5e5da35 Merge: 5c89d5a 4c45e95 Author: Yasuo Ohgaki yohg...@php.net Date: Tue Oct 29 17:30:58 2013 +0900 Merge branch 'PHP-5.5' * PHP-5.5: Fixed Bug 64760 var_export() does not use full precision for floating-point numbers Although I don't strongly insist merging fix to 5.6 branch, I think data exchange function that is not trying to be precise is bad thing. when transferring data . If you change it, then it's screwed up because it will use different ini. We don't know what people do in their code and we should not break it. As I said this is not a bug but we could consider changing that if the RFC proposing such change passes. Strong json numeric validator may have problem with the change. However, JSON RFC states 6. Numbers This specification allows implementations to set limits on the range and precision of numbers accepted. Since software that implements IEEE 754-2008 binary64 (double precision) numbers [IEEE754] is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these provide, in the sense that implementations will approximate JSON numbers within the expected precision. A JSON number such as 1E400 or 3.141592653589793238462643383279 may indicate potential interoperability problems, since it suggests that the software that created it expects receiving software to have greater capabilities for numeric magnitude and precision than is widely available. Note that when such software is used, numbers that are integers and are in the range [-(2**53)+1, (2**53)-1] are interoperable in the sense that implementations will agree exactly on their numeric values. https://tools.ietf.org/html/rfc7159 In order PHP to be more compatible, we should use IEEE 754 range or our other data exchange standard, which is serialize_precision=17. My position is - PHP7.0: We must use serialize_precision for JSON - PHP5.6: I strongly suggest to use serialize_precision for JSON Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Yasuo, On Sun, Jul 26, 2015 at 9:20 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi Jakub, On Mon, Jul 27, 2015 at 3:32 AM, Jakub Zelenka bu...@php.net wrote: I don't think that this is a bug. Your example is also completely unrelated to json because the place when the value is rounded is var_dump where it's based on ini precision. You would get the same values with json_encode but it's only because it uses the same ini ( precision ). Basically it has nothing to do with json_decode. OK. Better example. [yohgaki@dev PHP-master]$ ./php-bin ?php $j = '{ v: 0.1234567890123456789 }'; var_dump(json_encode(json_decode($j))); ini_set('precision', 20); var_dump(json_encode(json_decode($j))); ? string(22) {v:0.12345678901235} string(28) {v:0.12345678901234567737} Yeah you will loose precision. That's just the fact that double can't contain such a big precision. This is of course the same for serialize and unserialize. See this example: http://3v4l.org/nb3mE The only difference is that that it is using different ini but you can still change it if you want. The precision lost is the same if the values are the same though. I see that that the bug report you created ( https://bugs.php.net/bug.php?id=70137 ) is actually about using serialize.precision instead of precision in json_encode . I agree that using 'precision' ini is not the best solution. However I don't think that start suddenly using serialize.precision is a good idea. First of all it's a big BC break that will be quite difficult to find (There is no way this should go to the point release because this is changing the generated json). Also the json encode usage is much wider than serialization. There might be use cases when you don't care about maximal precision and you prefer to save some space. Imagine that you are transferring lots of float numbers. Then it will result in increasing of the transfer size. It would be great if the new ini was used when it was introduced but I'm afraid that changing that now is not a good idea I think that due to BC break, this shouldn't be changed. If you really want a bigger precision, you can do ini_set('precision', 30) before json_encode and then set it back. It means that this not a bug because you can still change it if you want. That means that we can't change before PHP 8 or whatever comes after 7... :) Same argument could be done for serialize, but it's changed to keep more precise value. Manual recommend json_decode/encode for data serialization also. Warning Do not pass untrusted user input to unserialize(). Unserialization can result in code being loaded and executed due to object instantiation and autoloading, and a malicious user may be able to exploit this. Use a safe, standard data interchange format such as JSON (via json_decode() and json_encode()) if you need to pass serialized data to the user. http://jp2.php.net/manual/en/function.unserialize.php It does not make sense having different serialization for float. Losing effective precision is design bug. IMHO. As I said I think that using preicision constant was unfortunate but start using serialize_precision is not a win IMHO. Using serialize and json_encoding are not the same thing. In any case this is not a bug, it's just an unfortunate decision and we should not definitely change it in a bug fixing release. It's a BC break. Maybe if we do an RFC, it could be considered for 7.1 but voters will need to agree that. I'd imagine 3 voting choices: * keep it as it is (use precision) * use serialize_precision * introduce new json_precision Currently users cannot handle even Google Map JSON data correctly by default. Fixing this is just a matter of using serialize.precision. If users need larger setting than 17 for whatever reason, they may do ini_set('serialize.precision', 30) and the rest of codes are unaffected. you can still do ini_set('pricision', 17) resp. ini_set('pricision', 30). BTW, larger precision setting does not mean precise, but current behavior definitely lose effective values. Those who don't care for precise data handling wouldn't care for more precise data handling, do they? yeah but mind that the side effect of rising precision is an increasing size of the json_encoded data which is not what everyone is happy with... Cheers Jakub
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Yasuo, On Sun, Jul 26, 2015 at 4:20 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi Jakub, On Mon, Jul 27, 2015 at 3:32 AM, Jakub Zelenka bu...@php.net wrote: I don't think that this is a bug. Your example is also completely unrelated to json because the place when the value is rounded is var_dump where it's based on ini precision. You would get the same values with json_encode but it's only because it uses the same ini ( precision ). Basically it has nothing to do with json_decode. OK. Better example. [yohgaki@dev PHP-master]$ ./php-bin ?php $j = '{ v: 0.1234567890123456789 }'; var_dump(json_encode(json_decode($j))); ini_set('precision', 20); var_dump(json_encode(json_decode($j))); ? string(22) {v:0.12345678901235} string(28) {v:0.12345678901234567737} I think you missed the point. Parsing isn't dependent upon precision setting. Only dumping: http://3v4l.org/48VSt $j = '{ v: 0.1234567890123456789 }'; $d1 = json_decode($j); ini_set('precision', 20); $d2 = json_decode($j); var_dump($d1, $d2); //object(stdClass)#1 (1) { [v]= float(0.12345678901234567737) } //object(stdClass)#2 (1) { [v]= float(0.12345678901234567737) } Meaning that it's parsed correctly. There is no bug here. Anthony -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] json_decode/encode should return full precision values by default
On Tue, Jul 28, 2015 at 7:10 AM, Yasuo Ohgaki yohg...@ohgaki.net wrote: On Tue, Jul 28, 2015 at 6:54 AM, Anthony Ferrara ircmax...@gmail.com wrote: On Sun, Jul 26, 2015 at 4:20 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi Jakub, On Mon, Jul 27, 2015 at 3:32 AM, Jakub Zelenka bu...@php.net wrote: I don't think that this is a bug. Your example is also completely unrelated to json because the place when the value is rounded is var_dump where it's based on ini precision. You would get the same values with json_encode but it's only because it uses the same ini ( precision ). Basically it has nothing to do with json_decode. OK. Better example. [yohgaki@dev PHP-master]$ ./php-bin ?php $j = '{ v: 0.1234567890123456789 }'; var_dump(json_encode(json_decode($j))); ini_set('precision', 20); var_dump(json_encode(json_decode($j))); ? string(22) {v:0.12345678901235} string(28) {v:0.12345678901234567737} I think you missed the point. Parsing isn't dependent upon precision setting. Only dumping: http://3v4l.org/48VSt $j = '{ v: 0.1234567890123456789 }'; $d1 = json_decode($j); ini_set('precision', 20); $d2 = json_decode($j); var_dump($d1, $d2); //object(stdClass)#1 (1) { [v]= float(0.12345678901234567737) } //object(stdClass)#2 (1) { [v]= float(0.12345678901234567737) } Meaning that it's parsed correctly. There is no bug here. I understands PHP uses pure IEEE data and arithmetic and I disagree that this is not a bug. What I'm pointing it out is simple JSON operation truncates _valid/precise_ value. IEEE double can store 17 digit fraction part precisely, but PHP truncates it. ?php $j = '{ v: 0.1234567890123456789 }'; var_dump(json_encode(json_decode($j))); ini_set('precision', 20); var_dump(json_encode(json_decode($j))); ? string(22) {v:0.12345678901235} string(28) {v:0.12345678901234567737} The same problem in serialize/unserialize was fixed as a bug. Encode/decode should be as precise as possible by _default_. IMO. What's the point of truncation and having broken value? Get JSON data from Google maps and store the data using PHP, then users lose last 2 digits of fraction part by default. The value is changed and wrong. This is definitely a bug. We can write $old = ini_set('precision', 17); json_encode($var); ini_set('precision', $old); everywhere to workaround this problem. Question is Is this the way it should be?. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Anthony, On Tue, Jul 28, 2015 at 6:54 AM, Anthony Ferrara ircmax...@gmail.com wrote: On Sun, Jul 26, 2015 at 4:20 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi Jakub, On Mon, Jul 27, 2015 at 3:32 AM, Jakub Zelenka bu...@php.net wrote: I don't think that this is a bug. Your example is also completely unrelated to json because the place when the value is rounded is var_dump where it's based on ini precision. You would get the same values with json_encode but it's only because it uses the same ini ( precision ). Basically it has nothing to do with json_decode. OK. Better example. [yohgaki@dev PHP-master]$ ./php-bin ?php $j = '{ v: 0.1234567890123456789 }'; var_dump(json_encode(json_decode($j))); ini_set('precision', 20); var_dump(json_encode(json_decode($j))); ? string(22) {v:0.12345678901235} string(28) {v:0.12345678901234567737} I think you missed the point. Parsing isn't dependent upon precision setting. Only dumping: http://3v4l.org/48VSt $j = '{ v: 0.1234567890123456789 }'; $d1 = json_decode($j); ini_set('precision', 20); $d2 = json_decode($j); var_dump($d1, $d2); //object(stdClass)#1 (1) { [v]= float(0.12345678901234567737) } //object(stdClass)#2 (1) { [v]= float(0.12345678901234567737) } Meaning that it's parsed correctly. There is no bug here. I understands PHP uses pure IEEE data and arithmetic and I disagree that this is not a bug. What I'm pointing it out is simple JSON operation truncates _valid/precise_ value. IEEE double can store 17 digit fraction part precisely, but PHP truncates it. ?php $j = '{ v: 0.1234567890123456789 }'; var_dump(json_encode(json_decode($j))); ini_set('precision', 20); var_dump(json_encode(json_decode($j))); ? string(22) {v:0.12345678901235} string(28) {v:0.12345678901234567737} The same problem in serialize/unserialize was fixed as a bug. Encode/decode should be as precise as possible by _default_. IMO. What's the point of truncation and having broken value? Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi! We can write $old = ini_set('precision', 17); json_encode($var); ini_set('precision', $old); everywhere to workaround this problem. If we're talking about encoding, that's what precision setting is - it says how much we want to preserve in decimal encoding of float numbers. Decoding, of course, should use whatever data available and not truncate, but I understand that is already happening? Question is Is this the way it should be?. I don't see why not. That's exactly what precision setting is for, as far as I can see. -- Stas Malyshev smalys...@gmail.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi all, On Tue, Jul 28, 2015 at 9:34 AM, Stanislav Malyshev smalys...@gmail.com wrote: We can write $old = ini_set('precision', 17); json_encode($var); ini_set('precision', $old); everywhere to workaround this problem. If we're talking about encoding, that's what precision setting is - it says how much we want to preserve in decimal encoding of float numbers. Decoding, of course, should use whatever data available and not truncate, but I understand that is already happening? Question is Is this the way it should be?. I don't see why not. That's exactly what precision setting is for, as far as I can see. JSON is var exporter/importer like serialize/var_export. Both of serialize/var_export uses PG(serialize_precision), what's the point of _lose_ / _destroy_ original values while PHP could keep more precise values than now? Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi! JSON is var exporter/importer like serialize/var_export. Both of serialize/var_export uses PG(serialize_precision), what's the point of _lose_ / _destroy_ original values while PHP could keep more precise values than now? So you're saying JSON should use serialize_precision setting instead of precision setting? That makes sense, I think it would be a reasonable change. -- Stas Malyshev smalys...@gmail.com -- PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi Jakub, On Mon, Jul 27, 2015 at 3:32 AM, Jakub Zelenka bu...@php.net wrote: I don't think that this is a bug. Your example is also completely unrelated to json because the place when the value is rounded is var_dump where it's based on ini precision. You would get the same values with json_encode but it's only because it uses the same ini ( precision ). Basically it has nothing to do with json_decode. OK. Better example. [yohgaki@dev PHP-master]$ ./php-bin ?php $j = '{ v: 0.1234567890123456789 }'; var_dump(json_encode(json_decode($j))); ini_set('precision', 20); var_dump(json_encode(json_decode($j))); ? string(22) {v:0.12345678901235} string(28) {v:0.12345678901234567737} I see that that the bug report you created ( https://bugs.php.net/bug.php?id=70137 ) is actually about using serialize.precision instead of precision in json_encode . I agree that using 'precision' ini is not the best solution. However I don't think that start suddenly using serialize.precision is a good idea. First of all it's a big BC break that will be quite difficult to find (There is no way this should go to the point release because this is changing the generated json). Also the json encode usage is much wider than serialization. There might be use cases when you don't care about maximal precision and you prefer to save some space. Imagine that you are transferring lots of float numbers. Then it will result in increasing of the transfer size. It would be great if the new ini was used when it was introduced but I'm afraid that changing that now is not a good idea I think that due to BC break, this shouldn't be changed. If you really want a bigger precision, you can do ini_set('precision', 30) before json_encode and then set it back. It means that this not a bug because you can still change it if you want. That means that we can't change before PHP 8 or whatever comes after 7... :) Same argument could be done for serialize, but it's changed to keep more precise value. Manual recommend json_decode/encode for data serialization also. Warning Do not pass untrusted user input to unserialize(). Unserialization can result in code being loaded and executed due to object instantiation and autoloading, and a malicious user may be able to exploit this. Use a safe, standard data interchange format such as JSON (via json_decode() and json_encode()) if you need to pass serialized data to the user. http://jp2.php.net/manual/en/function.unserialize.php It does not make sense having different serialization for float. Losing effective precision is design bug. IMHO. Currently users cannot handle even Google Map JSON data correctly by default. Fixing this is just a matter of using serialize.precision. If users need larger setting than 17 for whatever reason, they may do ini_set('serialize.precision', 30) and the rest of codes are unaffected. BTW, larger precision setting does not mean precise, but current behavior definitely lose effective values. Those who don't care for precise data handling wouldn't care for more precise data handling, do they? Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] json_decode/encode should return full precision values by default
Hi, On Sat, Jul 25, 2015 at 11:01 PM, Yasuo Ohgaki yohg...@ohgaki.net wrote: Hi all, I had to work with Google Map API and need to handle values precisely. Fortunately, it seems values are IEEE double, but I get rounded float values by default. For example, ?php $json = ' { results : [ { elevation : 1608.637939453125, location : { lat : 39.73915360, lng : -104.98470340 }, resolution : 4.771975994110107 }, { elevation : -50.78903579711914, location : { lat : 36.460, lng : -116.870 }, resolution : 19.08790397644043 } ], status : OK } '; var_dump(json_decode($json)); ? object(stdClass)#5 (2) { [results]= array(2) { [0]= object(stdClass)#1 (3) { [elevation]= float(1608.6379394531) [location]= object(stdClass)#2 (2) { [lat]= float(39.7391536) [lng]= float(-104.9847034) } [resolution]= float(4.7719759941101) } [1]= object(stdClass)#3 (3) { [elevation]= float(-50.789035797119) [location]= object(stdClass)#4 (2) { [lat]= float(36.46) [lng]= float(-116.87) } [resolution]= float(19.08790397644) } } [status]= string(2) OK } json_decode()/json_encode() must be able to handle precise IEEE double value by _default_. serialize() is changed to use max precision. json_decode/encode should do the same at least. I think this change should be provided as bug fix. Any comments? I don't think that this is a bug. Your example is also completely unrelated to json because the place when the value is rounded is var_dump where it's based on ini precision. You would get the same values with json_encode but it's only because it uses the same ini ( precision ). Basically it has nothing to do with json_decode. I see that that the bug report you created ( https://bugs.php.net/bug.php?id=70137 ) is actually about using serialize.precision instead of precision in json_encode . I agree that using 'precision' ini is not the best solution. However I don't think that start suddenly using serialize.precision is a good idea. First of all it's a big BC break that will be quite difficult to find (There is no way this should go to the point release because this is changing the generated json). Also the json encode usage is much wider than serialization. There might be use cases when you don't care about maximal precision and you prefer to save some space. Imagine that you are transferring lots of float numbers. Then it will result in increasing of the transfer size. It would be great if the new ini was used when it was introduced but I'm afraid that changing that now is not a good idea I think that due to BC break, this shouldn't be changed. If you really want a bigger precision, you can do ini_set('precision', 30) before json_encode and then set it back. It means that this not a bug because you can still change it if you want. That means that we can't change before PHP 8 or whatever comes after 7... :) Cheers Jakub
[PHP-DEV] json_decode/encode should return full precision values by default
Hi all, I had to work with Google Map API and need to handle values precisely. Fortunately, it seems values are IEEE double, but I get rounded float values by default. For example, ?php $json = ' { results : [ { elevation : 1608.637939453125, location : { lat : 39.73915360, lng : -104.98470340 }, resolution : 4.771975994110107 }, { elevation : -50.78903579711914, location : { lat : 36.460, lng : -116.870 }, resolution : 19.08790397644043 } ], status : OK } '; var_dump(json_decode($json)); ? object(stdClass)#5 (2) { [results]= array(2) { [0]= object(stdClass)#1 (3) { [elevation]= float(1608.6379394531) [location]= object(stdClass)#2 (2) { [lat]= float(39.7391536) [lng]= float(-104.9847034) } [resolution]= float(4.7719759941101) } [1]= object(stdClass)#3 (3) { [elevation]= float(-50.789035797119) [location]= object(stdClass)#4 (2) { [lat]= float(36.46) [lng]= float(-116.87) } [resolution]= float(19.08790397644) } } [status]= string(2) OK } json_decode()/json_encode() must be able to handle precise IEEE double value by _default_. serialize() is changed to use max precision. json_decode/encode should do the same at least. I think this change should be provided as bug fix. Any comments? -- Yasuo Ohgaki yohg...@ohgaki.net