Re: Re : Re: protbuf to json not compatible
As a note, if you just need the protobufs you should be able to run protoc against the published protos for the version you're building against without needing to pull in the 'mesos.native' package. The 'mesos.native' package provides scheduler and executor implementations (via talking to the C++ mesos libs), and it sounds like you just need the .proto files? I'm curious how other projects deal with "publishing" protobuf, whether they publish packages compiled by "protoc" per language or just publish the .protos. Seems like the latter is the more reasonable one for projects to do. On Sat, Mar 25, 2017 at 12:40 AM, Olivier Sallou wrote: > > - Benjamin Mahler a écrit : > > James, I'm curious, do you know specifically what the incompatibility is? > > > > Olivier, if you're dealing with protobuf already and trying to send it to > > mesos, there's no need to use JSON. Unless you have a requirement to do > so? > > I can manage json, this is fine. > Sending protobuf mean sending whole accept message as protobuf, not task > definition only. But for this I need mesos.native python package, and i > want to avoid this. So i will switch to full json. > > Olivier > > > > > There are some outstanding issues with our JSON<->Protobuf conversion, > > specifically we currently are inconsistent from proto3 when it comes to > the > > int(32|64), fixed(32|64), uint(32|64) handling, for one (we don't allow > > strings on the input side (tomek is addressing that), and we don't use > > strings on the output side). > > > > On Fri, Mar 24, 2017 at 12:44 AM, Olivier Sallou < > olivier.sal...@irisa.fr> > > wrote: > > > > > > > > > > > On 03/24/2017 04:02 AM, James Peach wrote: > > > >> On Mar 23, 2017, at 7:58 PM, James Peach wrote: > > > >> > > > >>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou < > olivier.sal...@irisa.fr> > > > wrote: > > > >>> > > > >>> Hi, > > > >>> > > > >>> when transforming a protobug message to json with MessageToJson, > the > > > >>> json is not compatible with the json format expected by Mesos > master. > > > >> This is because you generated the protobuf bindings with proto3 > > > compiler. AFAICT they made an incompatible change to the JSON wire > format. > > > This bites you when using the jsonpb Go package, for example. I ended > up > > > post-processing the generated Go code to correct the field names. > > > > Sorry I forgot to mention that the other workaround is to generate > the > > > protobuf bindings with the proto2 compiler. > > > Thanks > > > My first workaround is to generate json directly, not a big deal in my > > > case, but I wanted to understand. > > > > > > Olivier > > > > > > > > J > > > > > > -- > > > Olivier Sallou > > > IRISA / University of Rennes 1 > > > Campus de Beaulieu, 35000 RENNES - FRANCE > > > Tel: 02.99.84.71.95 > > > > > > gpg key id: 4096R/326D8438 (keyring.debian.org) > > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > > > > > > > >
Re : Re: protbuf to json not compatible
- Benjamin Mahler a écrit : > James, I'm curious, do you know specifically what the incompatibility is? > > Olivier, if you're dealing with protobuf already and trying to send it to > mesos, there's no need to use JSON. Unless you have a requirement to do so? I can manage json, this is fine. Sending protobuf mean sending whole accept message as protobuf, not task definition only. But for this I need mesos.native python package, and i want to avoid this. So i will switch to full json. Olivier > There are some outstanding issues with our JSON<->Protobuf conversion, > specifically we currently are inconsistent from proto3 when it comes to the > int(32|64), fixed(32|64), uint(32|64) handling, for one (we don't allow > strings on the input side (tomek is addressing that), and we don't use > strings on the output side). > > On Fri, Mar 24, 2017 at 12:44 AM, Olivier Sallou > wrote: > > > > > > > On 03/24/2017 04:02 AM, James Peach wrote: > > >> On Mar 23, 2017, at 7:58 PM, James Peach wrote: > > >> > > >>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou > > wrote: > > >>> > > >>> Hi, > > >>> > > >>> when transforming a protobug message to json with MessageToJson, the > > >>> json is not compatible with the json format expected by Mesos master. > > >> This is because you generated the protobuf bindings with proto3 > > compiler. AFAICT they made an incompatible change to the JSON wire format. > > This bites you when using the jsonpb Go package, for example. I ended up > > post-processing the generated Go code to correct the field names. > > > Sorry I forgot to mention that the other workaround is to generate the > > protobuf bindings with the proto2 compiler. > > Thanks > > My first workaround is to generate json directly, not a big deal in my > > case, but I wanted to understand. > > > > Olivier > > > > > > J > > > > -- > > Olivier Sallou > > IRISA / University of Rennes 1 > > Campus de Beaulieu, 35000 RENNES - FRANCE > > Tel: 02.99.84.71.95 > > > > gpg key id: 4096R/326D8438 (keyring.debian.org) > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > > > >
Re: protbuf to json not compatible
> On Mar 24, 2017, at 12:49 PM, Benjamin Mahler wrote: > > James, I'm curious, do you know specifically what the incompatibility is? https://developers.google.com/protocol-buffers/docs/proto3#json "Message field names are mapped to lowerCamelCase and become JSON object keys. " So field names like "failover_timeout" are renamed to "failoverTimeout". > > Olivier, if you're dealing with protobuf already and trying to send it to > mesos, there's no need to use JSON. Unless you have a requirement to do so? > There are some outstanding issues with our JSON<->Protobuf conversion, > specifically we currently are inconsistent from proto3 when it comes to the > int(32|64), fixed(32|64), uint(32|64) handling, for one (we don't allow > strings on the input side (tomek is addressing that), and we don't use > strings on the output side). > > On Fri, Mar 24, 2017 at 12:44 AM, Olivier Sallou > wrote: > >> >> >> On 03/24/2017 04:02 AM, James Peach wrote: On Mar 23, 2017, at 7:58 PM, James Peach wrote: > On Mar 23, 2017, at 1:54 AM, Olivier Sallou >> wrote: > > Hi, > > when transforming a protobug message to json with MessageToJson, the > json is not compatible with the json format expected by Mesos master. This is because you generated the protobuf bindings with proto3 >> compiler. AFAICT they made an incompatible change to the JSON wire format. >> This bites you when using the jsonpb Go package, for example. I ended up >> post-processing the generated Go code to correct the field names. >>> Sorry I forgot to mention that the other workaround is to generate the >> protobuf bindings with the proto2 compiler. >> Thanks >> My first workaround is to generate json directly, not a big deal in my >> case, but I wanted to understand. >> >> Olivier >>> >>> J >> >> -- >> Olivier Sallou >> IRISA / University of Rennes 1 >> Campus de Beaulieu, 35000 RENNES - FRANCE >> Tel: 02.99.84.71.95 >> >> gpg key id: 4096R/326D8438 (keyring.debian.org) >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >> >>
Re: protbuf to json not compatible
Tomek I think that patch also needs to address accepting strings for floating point fields: float, double. IMO, to avoid confusion, we want to get as close as possible to the proto3 specified mapping, which allows string inputs for all types of numeric fields: https://developers.google.com/protocol-buffers/docs/proto3#json This means that on the output side, we need to figure out how to transition to using strings for 64 bit integers without breaking our existing endpoint consumers. On Fri, Mar 24, 2017 at 12:31 PM, Anand Mazumdar wrote: > Hi Tomek, > > Looks like we dropped the ball on MESOS-5995 > (https://issues.apache.org/jira/browse/MESOS-5995). I assigned myself > as the shepherd and would take a look next week. > > -anand > > On Thu, Mar 23, 2017 at 2:09 AM, Tomek Janiszewski > wrote: > > I have a similar problem with protobuf and json. In my case numbers were > > incorrectly unmarshaled. Here is an issue > > https://issues.apache.org/jira/browse/MESOS-970 and review > > https://reviews.apache.org/r/50851/ > > > > czw., 23.03.2017, 09:54 użytkownik Olivier Sallou < > olivier.sal...@irisa.fr> > > napisał: > > > >> Hi, > >> > >> when transforming a protobug message to json with MessageToJson, the > >> json is not compatible with the json format expected by Mesos master. > >> > >> For example, for volumes it generates > >> > >> > >> volumes: [ > >> > >> {'hostPath': '', > >> > >> 'containerPath': '...', > >> > >> ... > >> > >>} > >> > >> ] > >> > >> > >> but HTTP API expects "source" and "container_path" > >> > >> is it an expected behavior ? This prevents from "creating" a task in > >> protobuf format and sending it to HTTP API with a protobug to json > >> conversion. > >> > >> Thanks > >> > >> Olivier > >> > >> -- > >> Olivier Sallou > >> IRISA / University of Rennes 1 > >> Campus de Beaulieu, 35000 RENNES - FRANCE > >> Tel: 02.99.84.71.95 > >> > >> gpg key id: 4096R/326D8438 (keyring.debian.org) > >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >> > >> >
Re: protbuf to json not compatible
James, I'm curious, do you know specifically what the incompatibility is? Olivier, if you're dealing with protobuf already and trying to send it to mesos, there's no need to use JSON. Unless you have a requirement to do so? There are some outstanding issues with our JSON<->Protobuf conversion, specifically we currently are inconsistent from proto3 when it comes to the int(32|64), fixed(32|64), uint(32|64) handling, for one (we don't allow strings on the input side (tomek is addressing that), and we don't use strings on the output side). On Fri, Mar 24, 2017 at 12:44 AM, Olivier Sallou wrote: > > > On 03/24/2017 04:02 AM, James Peach wrote: > >> On Mar 23, 2017, at 7:58 PM, James Peach wrote: > >> > >>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou > wrote: > >>> > >>> Hi, > >>> > >>> when transforming a protobug message to json with MessageToJson, the > >>> json is not compatible with the json format expected by Mesos master. > >> This is because you generated the protobuf bindings with proto3 > compiler. AFAICT they made an incompatible change to the JSON wire format. > This bites you when using the jsonpb Go package, for example. I ended up > post-processing the generated Go code to correct the field names. > > Sorry I forgot to mention that the other workaround is to generate the > protobuf bindings with the proto2 compiler. > Thanks > My first workaround is to generate json directly, not a big deal in my > case, but I wanted to understand. > > Olivier > > > > J > > -- > Olivier Sallou > IRISA / University of Rennes 1 > Campus de Beaulieu, 35000 RENNES - FRANCE > Tel: 02.99.84.71.95 > > gpg key id: 4096R/326D8438 (keyring.debian.org) > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >
Re: protbuf to json not compatible
Hi Tomek, Looks like we dropped the ball on MESOS-5995 (https://issues.apache.org/jira/browse/MESOS-5995). I assigned myself as the shepherd and would take a look next week. -anand On Thu, Mar 23, 2017 at 2:09 AM, Tomek Janiszewski wrote: > I have a similar problem with protobuf and json. In my case numbers were > incorrectly unmarshaled. Here is an issue > https://issues.apache.org/jira/browse/MESOS-970 and review > https://reviews.apache.org/r/50851/ > > czw., 23.03.2017, 09:54 użytkownik Olivier Sallou > napisał: > >> Hi, >> >> when transforming a protobug message to json with MessageToJson, the >> json is not compatible with the json format expected by Mesos master. >> >> For example, for volumes it generates >> >> >> volumes: [ >> >> {'hostPath': '', >> >> 'containerPath': '...', >> >> ... >> >>} >> >> ] >> >> >> but HTTP API expects "source" and "container_path" >> >> is it an expected behavior ? This prevents from "creating" a task in >> protobuf format and sending it to HTTP API with a protobug to json >> conversion. >> >> Thanks >> >> Olivier >> >> -- >> Olivier Sallou >> IRISA / University of Rennes 1 >> Campus de Beaulieu, 35000 RENNES - FRANCE >> Tel: 02.99.84.71.95 >> >> gpg key id: 4096R/326D8438 (keyring.debian.org) >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >> >>
Re: protbuf to json not compatible
On 03/24/2017 04:02 AM, James Peach wrote: >> On Mar 23, 2017, at 7:58 PM, James Peach wrote: >> >>> On Mar 23, 2017, at 1:54 AM, Olivier Sallou wrote: >>> >>> Hi, >>> >>> when transforming a protobug message to json with MessageToJson, the >>> json is not compatible with the json format expected by Mesos master. >> This is because you generated the protobuf bindings with proto3 compiler. >> AFAICT they made an incompatible change to the JSON wire format. This bites >> you when using the jsonpb Go package, for example. I ended up >> post-processing the generated Go code to correct the field names. > Sorry I forgot to mention that the other workaround is to generate the > protobuf bindings with the proto2 compiler. Thanks My first workaround is to generate json directly, not a big deal in my case, but I wanted to understand. Olivier > > J -- Olivier Sallou IRISA / University of Rennes 1 Campus de Beaulieu, 35000 RENNES - FRANCE Tel: 02.99.84.71.95 gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
Re: protbuf to json not compatible
> On Mar 23, 2017, at 7:58 PM, James Peach wrote: > >> >> On Mar 23, 2017, at 1:54 AM, Olivier Sallou wrote: >> >> Hi, >> >> when transforming a protobug message to json with MessageToJson, the >> json is not compatible with the json format expected by Mesos master. > > This is because you generated the protobuf bindings with proto3 compiler. > AFAICT they made an incompatible change to the JSON wire format. This bites > you when using the jsonpb Go package, for example. I ended up post-processing > the generated Go code to correct the field names. Sorry I forgot to mention that the other workaround is to generate the protobuf bindings with the proto2 compiler. J
Re: protbuf to json not compatible
> On Mar 23, 2017, at 1:54 AM, Olivier Sallou wrote: > > Hi, > > when transforming a protobug message to json with MessageToJson, the > json is not compatible with the json format expected by Mesos master. This is because you generated the protobuf bindings with proto3 compiler. AFAICT they made an incompatible change to the JSON wire format. This bites you when using the jsonpb Go package, for example. I ended up post-processing the generated Go code to correct the field names. > > For example, for volumes it generates > > > volumes: [ > >{'hostPath': '', > > 'containerPath': '...', > > ... > > } > > ] > > > but HTTP API expects "source" and "container_path" > > is it an expected behavior ? This prevents from "creating" a task in > protobuf format and sending it to HTTP API with a protobug to json > conversion. It’s expected from a proto3 compiler. IMHO this is a breaking change and they should have made a fallback option, but there’s not one. There’s no good choices here AFAICT. J
Re: protbuf to json not compatible
+anand, vinod On Thu, Mar 23, 2017 at 2:09 AM, Tomek Janiszewski wrote: > I have a similar problem with protobuf and json. In my case numbers were > incorrectly unmarshaled. Here is an issue > https://issues.apache.org/jira/browse/MESOS-970 and review > https://reviews.apache.org/r/50851/ > > czw., 23.03.2017, 09:54 użytkownik Olivier Sallou > > napisał: > > > Hi, > > > > when transforming a protobug message to json with MessageToJson, the > > json is not compatible with the json format expected by Mesos master. > > > > For example, for volumes it generates > > > > > > volumes: [ > > > > {'hostPath': '', > > > > 'containerPath': '...', > > > > ... > > > >} > > > > ] > > > > > > but HTTP API expects "source" and "container_path" > > > > is it an expected behavior ? This prevents from "creating" a task in > > protobuf format and sending it to HTTP API with a protobug to json > > conversion. > > > > Thanks > > > > Olivier > > > > -- > > Olivier Sallou > > IRISA / University of Rennes 1 > > Campus de Beaulieu, 35000 RENNES - FRANCE > > Tel: 02.99.84.71.95 > > > > gpg key id: 4096R/326D8438 (keyring.debian.org) > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > > > > >
Re: protbuf to json not compatible
+gilbert On Thu, Mar 23, 2017 at 1:54 AM, Olivier Sallou wrote: > Hi, > > when transforming a protobug message to json with MessageToJson, the > json is not compatible with the json format expected by Mesos master. > > For example, for volumes it generates > > > volumes: [ > > {'hostPath': '', > > 'containerPath': '...', > > ... > >} > > ] > > > but HTTP API expects "source" and "container_path" > > is it an expected behavior ? This prevents from "creating" a task in > protobuf format and sending it to HTTP API with a protobug to json > conversion. > > Thanks > > Olivier > > -- > Olivier Sallou > IRISA / University of Rennes 1 > Campus de Beaulieu, 35000 RENNES - FRANCE > Tel: 02.99.84.71.95 > > gpg key id: 4096R/326D8438 (keyring.debian.org) > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >
Re: protbuf to json not compatible
I have a similar problem with protobuf and json. In my case numbers were incorrectly unmarshaled. Here is an issue https://issues.apache.org/jira/browse/MESOS-970 and review https://reviews.apache.org/r/50851/ czw., 23.03.2017, 09:54 użytkownik Olivier Sallou napisał: > Hi, > > when transforming a protobug message to json with MessageToJson, the > json is not compatible with the json format expected by Mesos master. > > For example, for volumes it generates > > > volumes: [ > > {'hostPath': '', > > 'containerPath': '...', > > ... > >} > > ] > > > but HTTP API expects "source" and "container_path" > > is it an expected behavior ? This prevents from "creating" a task in > protobuf format and sending it to HTTP API with a protobug to json > conversion. > > Thanks > > Olivier > > -- > Olivier Sallou > IRISA / University of Rennes 1 > Campus de Beaulieu, 35000 RENNES - FRANCE > Tel: 02.99.84.71.95 > > gpg key id: 4096R/326D8438 (keyring.debian.org) > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > >
protbuf to json not compatible
Hi, when transforming a protobug message to json with MessageToJson, the json is not compatible with the json format expected by Mesos master. For example, for volumes it generates volumes: [ {'hostPath': '', 'containerPath': '...', ... } ] but HTTP API expects "source" and "container_path" is it an expected behavior ? This prevents from "creating" a task in protobuf format and sending it to HTTP API with a protobug to json conversion. Thanks Olivier -- Olivier Sallou IRISA / University of Rennes 1 Campus de Beaulieu, 35000 RENNES - FRANCE Tel: 02.99.84.71.95 gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438