[issue23123] Only READ support for Decimal in json
Changes by Ethan Furman et...@stoneleaf.us: -- nosy: -ethan.furman ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Mark Dickinson added the comment: See also #16535. -- nosy: +mark.dickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Antoine Pitrou added the comment: To cope with this potential problem, compliant parsers must preserve the original textual representation of properties internally in order to support JCS normalization requirements That sounds ridiculous. Did someone try to reason the IETF guys? :) -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment: Antoine Pitrou added the comment: To cope with this potential problem, compliant parsers must preserve the original textual representation of properties internally in order to support JCS normalization requirements That sounds ridiculous. Did someone try to reason the IETF guys?:) The alternative is either doing what Bob suggested which is almost the same as writing a new parser or take the IETF route and shroud the message payload in base64. So all solutions are by definition bd :-) FWIW my super-bad solution has the following compatibility issues: - Whitespace: None, all parsers can stringify, right? - Escaping: None, all parsers MUST do it to follow the JSON spec. - Property order: A problem in some parsers. If you take a look on stackoverflow lots of folks request that insertion/reader order should be honored since computers humans. Fixed in Python. Works in browsers as well. - Floating point: an almost useless JSON feature anyway, it doesn't work for crypto-numbers or money. It is only a validation problem though. Now fixed in Python. http://www.ietf.org/mail-archive/web/acme/current/msg00200.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Antoine Pitrou added the comment: I won't claim to know/understand the specifics, but message payload in base64 actually sounds reasonable to me, if far from optimal (both from readibility and space overhead POV) :-). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment:
Antoine Pitrou added the comment:
I won't claim to know/understand the specifics, but message payload in
base64 actually sounds reasonable to me, if far from optimal (both from
readibility and space overhead POV) :-).
It is indeed a working solution. I do though think that communities that
previously used XML would accept base64-encoded messages. It becomes really
messy when applied to counter-signed messages like the following:
{
@context: http://xmlns.webpki.org/wcpp-payment-demo;,
@qualifier: AuthData,
paymentRequest:
{
commonName: Demo Merchant,
amount: 8600550,
currency: USD,
referenceId: #100,
dateTime: 2014-12-18T13:39:35Z,
signature:
{
algorithm: RS256,
signerCertificate:
{
issuer: CN=Merchant Network Sub CA5,C=DE,
serialNumber: 1413983542582,
subject: CN=Demo Merchant,2.5.4.5=#1306383936333235,C=DE
},
certificatePath:
[
MIIDQzCCAiugAwIBAgIGAUk3_J02M...eMGlY734U3NasQfAhTUhxrdDbphEvsWTc,
MIIEPzCCAiegAwIBAgIBBTANBgkqh...gU1IyRGA7IbdHOeDB2RUpsXloU2QKfLrk
],
value:
Ny4Qe6FQhd5_qcSc3xiH8Kt7tIZ9Z...9LEjC6_Rulg_G20fGxJ-wzezFpsAGbmuFQg
}
},
domainName: merchant.com,
cardType: SuperCard,
pan: 1618342124919252,
dateTime: 2014-12-18T13:40:59Z,
signature:
{
algorithm: RS256,
signerCertificate:
{
issuer: CN=Mybank Client Root CA1,C=US,
serialNumber: 1413983550045,
subject: CN=The Cardholder,2.5.4.5=#13083935363733353232
},
certificatePath:
[MIIENzCCAh-gAwIBAgIGAUk3_LpdM...IGcN1md5feo5DndNnV8D0UM-oBRkUDDFiWlhCU],
value:
wyUcFcHmvM5ZozZKOEwOQkYic0D7M...S_HbaPGau5KfZjCaksvb5U1lYZaXxP8kAbuGPQ
}
}
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23123
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Raymond Hettinger added the comment: Given that json is multi-language format ... I don't know that we can expect much more from it. JSON specifies a textual number format but doesn't dictate whether that format represents a fixed precision binary float point number or a decimal floating point number. It is perfectly reasonable for someone to want to read and write a JSON number format to and from a decimal (we also see this with database formats as well -- such as sqlite). This bug report isn't a JSON spec issue; rather, it is about how the JSON module API can support (or inhibit) valid use cases. AFAICT, the patch to make the API better support enums had the side-effect of inhibiting the APIs ability to support number objects that want to control their output via __str__ or __repr__. This seems to block-off decimal support and support for controlling displayed precision. I think the Enum patch is suspect and could be considered a regression. That said, we could simply add direct support for decimals and leave the enum patch in-place (though it still impairs a user's ability to control the displayed precision). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment: I guess my particular requirement/wish is unusual (keeping the original textual representation of a floating point number intact) while using Decimal should be fairly universal. If these things could be combined in a Decimal support option I would (of course) be extremely happy. They do not appear to be in conflict. Currently I'm a bit bogged down by the crypto-stuff since it is spread over different and incompatible modules which makes it awkward creating a nice unified RSA/EC solution. I may end-up writing a wrapper... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Bob Ippolito added the comment: I don't think it's reasonable to expect Decimal to always output precisely the same string it was given. It's a waste of complexity and space and the only time you would want this behavior is when you really should've left it accessible as a string in the first place. It sounds like the spec for that signature may be poorly designed (with regard to portability). Relying on the precise string output of a number is not going to work in any JSON parser I've ever seen. You'd need to work at the tokenizer level and not all of the parsers provide an interface at that layer (since many of them combine tokenization and parsing). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment: Bob, Your'e right, I have put up a requirement for JSON serializing that may be over the top. OTOH, there are (AFAICT...) only two possible solutions: 1. Outlaw floating point data from the plot 2. Insist that serializers conform to the spec As a pragmatic I have settled on something in between :-) https://openkeystore.googlecode.com/svn/resources/trunk/docs/jcs.html#Interoperability I don't think that the overhead in Decimal would be a problem but I'm not a Python platform maintainer so I leave it to you guys. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment: Well, I could have insisted on canonicalization of floating-point data but that's so awkward that outlawing such data is a cleaner approach. Since the target for JCS is security- and payment-protocols, I don't think the absence of floating-point support will be a show-stopper. I does though make the IETF folks unhappy. Another reason for still wanting it to work as currently specified is because it would be nice to have JCS running on three fully compatible platforms, including one which I haven't designed :-) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment:
Using simplejson I got it to work!!!
I just wonder what you think of the solution:
import collections
import simplejson as json
from decimal import Decimal
class EnhancedDecimal(Decimal):
def __str__ (self):
return self.saved_string
def __new__(cls, value=0, context=None):
obj = Decimal.__new__(cls,value,context)
obj.saved_string = value
return obj;
jsonString = '{t:6,h:4.50, g:text,j:1.40e450}'
jsonObject = json.loads(jsonString,
object_pairs_hook=collections.OrderedDict,parse_float=EnhancedDecimal)
for item in jsonObject:
print jsonObject[item]
print json.dumps(jsonObject)
6
4.50
text
1.40e450
{t: 6, h: 4.50, g: text, j: 1.40e450}
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23123
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Bob Ippolito added the comment: Yeah, that's the hack I was suggesting. I suppose I don't see the point of having a protocol that normalizes *almost* everything. Normalization should be all or nothing. Other options would be to define the signature at the encoded byte level with no normalization (in which case you could use any off the shelf signing), or at the value level and prescribe a specific interpretation for data types. I would've done it at the value level and prescribed that dictionaries should be key sorted, strings dealt with as UTF-8, and numbers as IEEE 754. I would make sure not to depend on the decimal conversion of numbers, and just work with the serialized bit representation in a particular endian (which you can even do efficiently in modern browser JS with Float64Array, DataView and ArrayBuffer). For JS portability it'd probably treat *all* numbers as floats in the same way, whether they had a decimal to begin with or not. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment: Bob, I'm not sure I understand why you say that JCS requires *almost* full normalization. Using browsers you can generate fully compliant JCS objects using like 20 lines of javascript/webcrypto (here excluding base64 support). No normalization step is needed. But sure, the IETF JOSE WG has taken an entirely different approach and require JSON objects to be serialized and Base64-encoded. Then the Base64 is signed. Boring. And in conflict with complex messaging like: https://openkeystore.googlecode.com/svn/wcpp-payment-demo/trunk/docs/messages.html#UserAuthorizesTransaction Thanx anyway, I'm pretty happy with how it works now! Well, if Decimal didn't manipulate its argument I would be even happier :-) because then there wouldn't even be a hack. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment: The current JCS validator is only 150 lines and does both RSA and EC signatures: https://code.google.com/p/openkeystore/source/browse/python/trunk/src/org/webpki/json/JCSValidator.py My Java-version is much more advanced but this is quite useful anyway -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Ethan Furman added the comment: Raymond Hettinger added the comment: --- This bug report isn't a JSON spec issue; rather, it is about how the JSON module API can support (or inhibit) valid use cases. AFAICT, the patch to make the API better support enums had the side-effect of inhibiting the APIs ability to support number objects that want to control their output via __str__ or __repr__. This seems to block-off decimal support and support for controlling displayed precision. I think the Enum patch is suspect and could be considered a regression. That said, we could simply add direct support for decimals and leave the enum patch in-place (though it still impairs a user's ability to control the displayed precision). The enum patch is in issue18264 if anyone wants to read the discussion. I am not a regular json user, but my impression is the format is pretty basic, and we would be overloading it to try and keep numbers with three decimal places as Decimal, and anything else as float. Isn't json's main purpose to support data exchange between different programs of different languages? Not between different Python programs? -- nosy: +barry, eli.bendersky, ncoghlan ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment: Ethan Furman added the comment: I am not a regular json user, but my impression is the format is pretty basic, and we would be overloading it to try and keep numbers with three decimal places as Decimal, and anything else as float. Isn't json's main purpose to support data exchange between different programs of different languages? Not between different Python programs? Right, unfortunately the need to support non-native data types like big decimals, dates and blobs have lead to a certain amount of confusion and innovation among JSON tool designers. I (FWIW) do actually NOT want to extend a single bit from the RFC, I just want serializing to be non-invasive. If the parse_float option stays as is it seems that both the people who want big (non-standard) numbers and I who want somewhat non-standard serialization would be happy. I.e. a documentation snippet would be sufficient as far as I can tell. Serialization order of objects is apparently a hot topic https://code.google.com/p/v8/issues/detail?id=164 but Python has no problem with that. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Raymond Hettinger added the comment: It is unfortunate that there doesn't seem to be a way to round-trip Decimals. That would seem to be a fundamental capability that we should expect to support. I have a vague recollection that you used to be able to trick the encoder by returning a subclass of float with a custom __str__; however, I don't think that hack would work anymore because float subclasses now get coerced back to a regular float in order to make the json module work with Enums (which have a __str__ that is meaningless in JSON). In Python 3.5, it would be nice to add a hook that affords more control than cls currently does. Ideally, it should allow any class to special exactly what it wants written-out. Another option, for Py2.7, 3.4, and 3.5 is to add direct support for decimal instances (much like the enum support was backported). -- assignee: - bob.ippolito nosy: +bob.ippolito, ethan.furman, rhettinger versions: +Python 3.4, Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Ethan Furman added the comment: Enums (and other numeric subclasses), do not round-trip back to themselves. An IntEnum with the value of 4 is written as 4 and converted back from json as the integer 4 (not Settings.TabSpaces, or whatever). Given that json is multi-language format (or a javascript format -- but either way not a Python-specific format) I don't know that we can expect much more from it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Bob Ippolito added the comment: simplejson has had a use_decimal flag for output since 2.1.0 and has been enabled by default since 2.2.0. simplejson 3.2.0 introduced a for_json argument that checks objects for a method of that name for serialization. https://github.com/simplejson/simplejson/blob/master/CHANGES.txt -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment: I was actually hoping to implement the final part of this: https://openkeystore.googlecode.com/svn/resources/trunk/docs/jcs.html#Normalization_and_Signature_Validation It seems that the current Decimal implementation wouldn't save me anyway since it modifies the input :-( Anyway, floats in JSON have rather little use so maybe my existing Pyhton (PoC) solution will be good enough: https://code.google.com/p/openkeystore/source/browse/python/trunk/src/org/webpki/json/JCSValidator.py -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Bob Ippolito added the comment: I'm sure there's some hack that would allow you to preserve the input. I would try using parse_float and have it return some object that preserves the string and will be output in precisely the same way. It may need to be a Decimal subclass. I'm traveling for the next few weeks so I won't have much of a chance to investigate myself. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Anders Rundgren added the comment: It would be great if I could use a sub-classed Decimal during parsing but since it doesn't appear to be a way to serialize the result using the json package I'm probably stuck with the current 99% solution. I have solved this in Java and JavaScript by writing my own JSON stuff http://webpki.org/papers/keygen2/doc/org/webpki/json/package-summary.html but that method obviously doesn't scale and I'm a real n00b when it comes to Python although it was more fun than I had expected :-) A minor patch addressing serialization of Decimal would probably do fine (after sub-classing) and would be generally useful. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
Bob Ippolito added the comment: Subclass Decimal and implement __str__ to return your own representation. Use parse_float to use your Decimal subclass. Should work with simplejson, a similar hack may be possible with the json module. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue23123 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23123] Only READ support for Decimal in json
New submission from Anders Rundgren:
jsonString = '{t:6,h:4.50, g:text,j:1.40e450}'
jsonObject = json.loads(jsonString,
object_pairs_hook=collections.OrderedDict,parse_float=Decimal)
for item in jsonObject:
print jsonObject[item]
6
4.50
text
1.40E+450
Works as expected.
However, there seems to be no way to get back to the original JSON string as
far as I can tell since you have to convert Decimal to str in cls when using
json.dumps which adds around the arguments
--
components: Extension Modules
messages: 233139
nosy: anders.rundgren@gmail.com
priority: normal
severity: normal
status: open
title: Only READ support for Decimal in json
type: behavior
versions: Python 2.7
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue23123
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
