[jira] [Commented] (MESOS-3345) Expand the range of integer precision when converting into/out of json.
[
https://issues.apache.org/jira/browse/MESOS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909928#comment-14909928
]
Joris Van Remoortere commented on MESOS-3345:
-
{code}
commit 3396a8e7919f4d2f44e3bef30e9838cc30ff9b4f
Author: Alex Clemmer
Date: Sun Sep 27 15:41:17 2015 -0700
CMake: Updated CMake config to build Mesos against picojson v1.3.0.
Review: https://reviews.apache.org/r/38531
commit a051fde5413f660aa6c32fd33a6c3687bb905687
Author: Alex Clemmer
Date: Sun Sep 27 15:41:15 2015 -0700
CMake: Added preprocessor definitions required to build picojson 1.3.0.
Review: https://reviews.apache.org/r/38530
commit 1eb0519eb1ffefb388e9ecad36e4b24413febcb8
Author: Alex Clemmer
Date: Sun Sep 27 15:41:14 2015 -0700
CMake: Only compile proc_tests.cpp for Linux platforms.
Review: https://reviews.apache.org/r/38529
{code}
> Expand the range of integer precision when converting into/out of json.
> ---
>
> Key: MESOS-3345
> URL: https://issues.apache.org/jira/browse/MESOS-3345
> Project: Mesos
> Issue Type: Task
> Components: stout
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
> Fix For: 0.25.0
>
>
> For [MESOS-3299], we added some protobufs to represent time with integer
> precision. However, this precision is not maintained through protobuf <->
> JSON conversion, because of how our JSON encoders/decoders convert numbers to
> floating point.
> To maintain precision, we can try one of the following:
> * Try using a {{long double}} to represent a number.
> * Add logic to stringify/parse numbers without loss when possible.
> * Try representing {{int64_t}} as a string and parse it as such?
> * Update PicoJson and add a compiler flag, i.e. {{-DPICOJSON_USE_INT64}}
> In all cases, we'll need to make sure that:
> * Integers are properly stringified without loss.
> * The JSON decoder parses the integer without loss.
> * We have some unit tests for big (close to {{INT32_MAX}}/{{INT64_MAX}}) and
> small integers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3345) Expand the range of integer precision when converting into/out of json.
[
https://issues.apache.org/jira/browse/MESOS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14791077#comment-14791077
]
Joseph Wu commented on MESOS-3345:
--
With regards to supporting unsigned values between {{INT64_MAX}} and
{{UINT64_MAX}}:
* We can patch PicoJSON to parse those values as {{uint64_t}}.
* We can investigate using another parsing library.
* If we want extra precision beyond 64 or 80 bits per double, one possibility
is the [GMP library|https://gmplib.org/]
> Expand the range of integer precision when converting into/out of json.
> ---
>
> Key: MESOS-3345
> URL: https://issues.apache.org/jira/browse/MESOS-3345
> Project: Mesos
> Issue Type: Task
> Components: stout
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> For [MESOS-3299], we added some protobufs to represent time with integer
> precision. However, this precision is not maintained through protobuf <->
> JSON conversion, because of how our JSON encoders/decoders convert numbers to
> floating point.
> To maintain precision, we can try one of the following:
> * Try using a {{long double}} to represent a number.
> * Add logic to stringify/parse numbers without loss when possible.
> * Try representing {{int64_t}} as a string and parse it as such?
> * Update PicoJson and add a compiler flag, i.e. {{-DPICOJSON_USE_INT64}}
> In all cases, we'll need to make sure that:
> * Integers are properly stringified without loss.
> * The JSON decoder parses the integer without loss.
> * We have some unit tests for big (close to {{INT32_MAX}}/{{INT64_MAX}}) and
> small integers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3345) Expand the range of integer precision when converting into/out of json.
[
https://issues.apache.org/jira/browse/MESOS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14791202#comment-14791202
]
Joseph Wu commented on MESOS-3345:
--
The followup issue is linked: https://issues.apache.org/jira/browse/MESOS-3449
> Expand the range of integer precision when converting into/out of json.
> ---
>
> Key: MESOS-3345
> URL: https://issues.apache.org/jira/browse/MESOS-3345
> Project: Mesos
> Issue Type: Task
> Components: stout
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> For [MESOS-3299], we added some protobufs to represent time with integer
> precision. However, this precision is not maintained through protobuf <->
> JSON conversion, because of how our JSON encoders/decoders convert numbers to
> floating point.
> To maintain precision, we can try one of the following:
> * Try using a {{long double}} to represent a number.
> * Add logic to stringify/parse numbers without loss when possible.
> * Try representing {{int64_t}} as a string and parse it as such?
> * Update PicoJson and add a compiler flag, i.e. {{-DPICOJSON_USE_INT64}}
> In all cases, we'll need to make sure that:
> * Integers are properly stringified without loss.
> * The JSON decoder parses the integer without loss.
> * We have some unit tests for big (close to {{INT32_MAX}}/{{INT64_MAX}}) and
> small integers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3345) Expand the range of integer precision when converting into/out of json.
[
https://issues.apache.org/jira/browse/MESOS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14726446#comment-14726446
]
Joseph Wu commented on MESOS-3345:
--
I uploaded an experimental implementation which updates the PicoJSON library to
version 1.3.0.
By using their {{int64_t}} support, we don't need to break the Protobuf->JSON
mapping. There are still some limitations.
For example, you can't convert unsigned longs of greater than {{INT64_MAX}},
because of how PicoJSON does the conversion.
Reviews:
https://reviews.apache.org/r/38028/
https://reviews.apache.org/r/38030/
https://reviews.apache.org/r/38031/
> Expand the range of integer precision when converting into/out of json.
> ---
>
> Key: MESOS-3345
> URL: https://issues.apache.org/jira/browse/MESOS-3345
> Project: Mesos
> Issue Type: Task
> Components: stout
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> For [MESOS-3299], we added some protobufs to represent time with integer
> precision. However, this precision is not maintained through protobuf <->
> JSON conversion, because of how our JSON encoders/decoders convert numbers to
> floating point.
> To maintain precision, we can try one of the following:
> * Try using a {{long double}} to represent a number.
> * Add logic to stringify/parse numbers without loss when possible.
> * Try representing {{int64_t}} as a string and parse it as such?
> * Update PicoJson and add a compiler flag, i.e. {{-DPICOJSON_USE_INT64}}
> In all cases, we'll need to make sure that:
> * Integers are properly stringified without loss.
> * The JSON decoder parses the integer without loss.
> * We have some unit tests for big (close to {{INT32_MAX}}/{{INT64_MAX}}) and
> small integers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3345) Expand the range of integer precision when converting into/out of json.
[
https://issues.apache.org/jira/browse/MESOS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724305#comment-14724305
]
Vinod Kone commented on MESOS-3345:
---
Looks like ResourceStatistics have uint64 fields, which will be breaking if we
change the Protobuf to JSON mapping :(
https://github.com/apache/mesos/blob/master/src/slave/monitor.cpp#L140
https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L704
> Expand the range of integer precision when converting into/out of json.
> ---
>
> Key: MESOS-3345
> URL: https://issues.apache.org/jira/browse/MESOS-3345
> Project: Mesos
> Issue Type: Task
> Components: stout
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> For [MESOS-3299], we added some protobufs to represent time with integer
> precision. However, this precision is not maintained through protobuf <->
> JSON conversion, because of how our JSON encoders/decoders convert numbers to
> floating point.
> To maintain precision, we can:
> 1) Try using a {{long double}} to represent a number.
> 2) Add logic to stringify/parse numbers without loss when possible.
> In all cases, we'll need to make sure that:
> * Integers are properly stringified without loss.
> * The JSON decoder parses the integer without loss.
> * We have some unit tests for big (close to {{INT32_MAX}}/{{INT64_MAX}}) and
> small integers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-3345) Expand the range of integer precision when converting into/out of json.
[
https://issues.apache.org/jira/browse/MESOS-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724630#comment-14724630
]
Benjamin Mahler commented on MESOS-3345:
Yes, we use proto2, I'm suggesting we mimic the proto3 <-> JSON conversion
(much like we did with base64 for bytes fields). Yes, picojson will parse it as
a string, that's the point :) Within our JSON->Protobuf conversion, we have to
numify accordingly.
> Expand the range of integer precision when converting into/out of json.
> ---
>
> Key: MESOS-3345
> URL: https://issues.apache.org/jira/browse/MESOS-3345
> Project: Mesos
> Issue Type: Task
> Components: stout
>Reporter: Joseph Wu
>Assignee: Joseph Wu
>Priority: Minor
> Labels: json, mesosphere, protobuf
>
> For [MESOS-3299], we added some protobufs to represent time with integer
> precision. However, this precision is not maintained through protobuf <->
> JSON conversion, because of how our JSON encoders/decoders convert numbers to
> floating point.
> To maintain precision, we can try one of the following:
> * Try using a {{long double}} to represent a number.
> * Add logic to stringify/parse numbers without loss when possible.
> * Try representing {{int64_t}} as a string and parse it as such?
> * Update PicoJson and add a compiler flag, i.e. {{-DPICOJSON_USE_INT64}}
> In all cases, we'll need to make sure that:
> * Integers are properly stringified without loss.
> * The JSON decoder parses the integer without loss.
> * We have some unit tests for big (close to {{INT32_MAX}}/{{INT64_MAX}}) and
> small integers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
