On Wed, Nov 28, 2012 at 11:43:19AM +0100, Mike Hearn wrote: > Peter is correct that there are a few degrees of freedom in protobuf > serialization, though far fewer than with JSON.
FWIW I re-read the specs again and turns out my memory was wrong. (I last looked at this about four months ago) Duplicated fields are handled in a defined manner, with the last field seen in the serialization being the one whose value is used. Again, repeated fields are treated as elements of a list, preserving order. It does raise the interesting question do the implementations that don't preserve order of unknown fields, preserve the order of multiple unknown fields, either repeated or not? > I'd like to think upstream would be open to resolving these > ambiguities. I gotta admit, I suspect they won't be that open. Protocol buffers was designed because Google needed a fast serialization method suitable for many different internal projects. Needing round-trip idempotence seems like a rare requirement to me, especially for internal use. > Re-serialization of an Invoice message in the Payment message is a > potential source of mistakes. There's no need to ever concatenate > these messages and alternative implementations that don't order > serialized fields by tag number are missing an important optimization, > so they could be fixed. The main issue is treatment of unknown fields. > If/when the Invoice message is extended with other fields that are > round-tripped through an old client, the data may get lost. JSON > doesn't help resolve that either, of course. There are a few > solutions: Well, actually you can take advantage of the message concatination ability of protocol buffers to extend a message by simply appending the new fields to the existing thus either defining new fields, or overriding old values as required. If you want to de-duplicate though you run into the problem all over again. On the other hand JSON handles this case fine too provided that your JSON implementation supports dictionary objects with arbitrary fields. Just use the object as is and the unknown fields will be re-serialized properly at the other end. Some implementations will have to be careful to handle collisions with existing keys in the namespace. (consider in Python what would happen if you mapped your object to a class instance, and the serialization included the key "__init__") That said, JSON is quite problematic with numbers. For instance, you have to be careful to keep integers represented as pure integers below what Javascript can handle, the maximum integer exactly representable in a double float, or the JSON won't be parsable in Javascript even if many other languages handle it fine. Protocol buffers is at least pretty explicit about what size integers are. > 1) Change the type of the Invoice field in Payment to be "bytes" and > set it to be the hash of the originally received binary Invoice > message. Downside, requires merchants to track all outstanding > invoices. > 2) Ask protobufs upstream to modify the spec/implementations so > ordering of unknown fields is specified. The Python implementation > could be extended to support them so Python implementors don't end up > with accidental message downgrades. > 3) Language of the spec could be changed to explicitly state that the > received Invoice may not be binary-identical to the one that was sent, > in the case of a client that incorrectly downgrades the message. Thus > you'd be expected to check what the Invoice was using merchant_data > which is opaque and could just be, eg, a database key on your own end. > 4) Instead of submitting the entire Invoice back to the merchant, just > the merchant_data could be in the Payment message. > > Of the four options I prefer the last. What is the use case for > resubmitting the entire invoice anyway? Even if protobufs are improved Note that I think the SignedInvoice message itself is broken, because protobuf implementations have no reason to guarantee that they can give you the serialized bytes of the Invoice sub-message. It's a quite specific use-case that isn't needed for pretty much anything but crypto. FWIW I took a quick look at the official API's, C++, Java and Python, and as far as I can tell none of them support accessing the binary serialization of a message field other than by re-serializing the message. Really the invoice field should be declared as bytes serialized_invoice, as inconvenient as that is to work with. > so handling of round-tripping new messages through old [Python] > clients is more rigorous, some implementors will probably convert the > protobuf objects into some internal forms for whatever reason (or > serialize them to a database, etc) and they're very likely to mess up > the handling of unknown fields when they do it. Since the Payment message includes an *untrusted* Invoice that the vendor needs to authenticate the whole invoice no matter what on Payment reception. In many cases that implies they have to keep some sort of database of "quotes" or similar anyway as the client can change anything they want otherwise. Again that leads back to the argument of why not just stick with the merchant_dat as you suggest, which will usually be some short invoice number attached to a database? A vendor that wants to operation a stateless invoicing system can just stuff a HMAC-protected serialized invoice into the merchant_data I guess you could use a mutable invoice field as a way of achieving some sort of negotiation protocol, but I think it's better to stick to the original concept of just ensuring that the user is really paying the right amount to the right address. -- 'peter'[:-1]@petertodd.org
signature.asc
Description: Digital signature
------------------------------------------------------------------------------ Keep yourself connected to Go Parallel: INSIGHTS What's next for parallel hardware, programming and related areas? Interviews and blogs by thought leaders keep you ahead of the curve. http://goparallel.sourceforge.net
_______________________________________________ Bitcoin-development mailing list Bitcoin-development@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bitcoin-development
