Re: Can we update protobuf's version on trunk?

2017-03-30 Thread Andrew Wang
Great. If y'all are satisfied, I am too. My only other request is that we shade PB even for the non-client JARs, since empirically there are a lot of downstream projects pulling in our server-side artifacts. On Thu, Mar 30, 2017 at 9:55 AM, Stack wrote: > On Thu, Mar 30, 2017 at 9:16 AM, Chris

Re: Can we update protobuf's version on trunk?

2017-03-30 Thread Stack
On Thu, Mar 30, 2017 at 9:16 AM, Chris Douglas wrote: > On Wed, Mar 29, 2017 at 4:59 PM, Stack wrote: > >> The former; an intermediate handler decoding, [modifying,] and > >> encoding the record without losing unknown fields. > >> > > > > I did not try this. Did you? Otherwise I can. > > Yeah, I

Re: Can we update protobuf's version on trunk?

2017-03-30 Thread Chris Douglas
On Wed, Mar 29, 2017 at 4:59 PM, Stack wrote: >> The former; an intermediate handler decoding, [modifying,] and >> encoding the record without losing unknown fields. >> > > I did not try this. Did you? Otherwise I can. Yeah, I did. Same format. -C >> This looks fine. -C >> >> > Thanks, >> > St.A

Re: Can we update protobuf's version on trunk?

2017-03-29 Thread Stack
On Wed, Mar 29, 2017 at 3:12 PM, Chris Douglas wrote: > On Wed, Mar 29, 2017 at 1:13 PM, Stack wrote: > > Is the below evidence enough that pb3 in proto2 syntax mode does not drop > > 'unknown' fields? (Maybe you want evidence that java tooling behaves the > > same?) > > I reproduced your exampl

Re: Can we update protobuf's version on trunk?

2017-03-29 Thread Chris Douglas
On Wed, Mar 29, 2017 at 1:13 PM, Stack wrote: > Is the below evidence enough that pb3 in proto2 syntax mode does not drop > 'unknown' fields? (Maybe you want evidence that java tooling behaves the > same?) I reproduced your example with the Java tooling, including changing some of the fields in t

Re: Can we update protobuf's version on trunk?

2017-03-29 Thread Stack
Is the below evidence enough that pb3 in proto2 syntax mode does not drop 'unknown' fields? (Maybe you want evidence that java tooling behaves the same?) To be clear, when we say proxy above, are we expecting that a pb message deserialized by a process down-the-line that happens to have a crimped

Re: Can we update protobuf's version on trunk?

2017-03-29 Thread Stack
On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang wrote: > > > > > If unknown fields are dropped, then applications proxying tokens and > > other > > >> data between servers will effectively corrupt those messages, unless > we > > >> make everything opaque bytes, which- absent the convenient, > prenom

Re: Can we update protobuf's version on trunk?

2017-03-28 Thread Allen Wittenauer
> On Mar 28, 2017, at 5:09 PM, Chris Douglas wrote: > > I haven't seen data identifying PB as a bottleneck, but the > non-x86/non-Linux and dev setup arguments may make this worthwhile. -C FWIW, we have the same problem with leveldbjni-all. (See the ASF PowerPC build logs) I keep meani

Re: Can we update protobuf's version on trunk?

2017-03-28 Thread Tsuyoshi Ozawa
> We can define protobuf's version by using syntax = "proto2" or syntax > = "proto3" in proto files with proto3 compiler. I think this is what > Stack mentioned as compatibility mode. I mean, "We can define protobuf's 'specification' 'version at compilation time by using syntax = "proto2""... - T

Re: Can we update protobuf's version on trunk?

2017-03-28 Thread Tsuyoshi Ozawa
> Stack mentioned a compatibility mode that uses the proto2 semantics. We can define protobuf's version by using syntax = "proto2" or syntax = "proto3" in proto files with proto3 compiler. I think this is what Stack mentioned as compatibility mode. https://github.com/golang/protobuf/blob/master/p

Re: Can we update protobuf's version on trunk?

2017-03-28 Thread Chris Douglas
On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang wrote: > Unfortunately, it sounds like these are intrinsic differences with PB3. That's too bad... but possibly not fatal: most of the data we proxy through client code is, if not opaque, it's at least immutable (particularly tokens). If PB3 does suppo

Re: Can we update protobuf's version on trunk?

2017-03-28 Thread Andrew Wang
> > > If unknown fields are dropped, then applications proxying tokens and > other > >> data between servers will effectively corrupt those messages, unless we > >> make everything opaque bytes, which- absent the convenient, prenominate > >> semantics managing the conversion- obviate the compatibil

Re: Can we update protobuf's version on trunk?

2017-03-28 Thread Chris Douglas
On Tue, Mar 28, 2017 at 3:04 PM, Andrew Wang wrote: > There's no mention of the convenient "Embedded messages are compatible with >> bytes if the bytes contain an encoded version of the message" semantics in >> proto3. > > > I checked the proto3 guide, and I think this is supported: > https://deve

Re: Can we update protobuf's version on trunk?

2017-03-28 Thread Andrew Wang
I've been investigating this a bit. I'm hoping Chris can ring in, since he's identified wire compatibility issues. Replying inline to Chris' comment

Re: Can we update protobuf's version on trunk?

2017-03-27 Thread Tsuyoshi Ozawa
Forwarding to common-dev, hdfs-dev, mapreduce-dev too. Thanks - Tsuyoshi 2017年3月27日(月) 21:16 Tsuyoshi Ozawa : > Dear Hadoop developers, > > After shaded client, introduced by HADOOP-11804, is merged, > we can more easily update some dependency with minimizing the impact > of backward compatibili

Can we update protobuf's version on trunk?

2017-03-27 Thread Tsuyoshi Ozawa
Dear Hadoop developers, After shaded client, introduced by HADOOP-11804, is merged, we can more easily update some dependency with minimizing the impact of backward compatibility on trunk. (Thanks Sean and Sanjin for taking the issue!) Then, is it time to update protobuf's version to the latest o