Thanks for that link, Alan. That looks like a useful site! Ideally, the Protocol Buffers project would give a clear statement about wire compatibility between 2.5.0 and 2.6.1. Unfortunately, I can't find that anywhere. If it's not documented, then it's probably worth following up on the Protocol Buffers support lists to ask them.
One thing we could try is starting up a mix of Hadoop processes using 2.5.0 and 2.6.1 to see how it goes. We've made a commitment to both forward and backward compatibility within Hadoop 2.x, so we'd need a 2.5.0 client to be able to talk to a 2.6.1 server, and we'd need a 2.6.1 client to be able to talk to a 2.5.0 server. Even if this appears to go well, I wouldn't consider it a substitute for a formal statement of the compatibility policy from the Protocol Buffers project. Otherwise, there might be some subtle lurking issue that we miss in our initial testing. As a reminder though, the community probably would want to see a strong justification for the upgrade in terms of features or performance or something else. Right now, I'm not seeing a significant benefit for us based on my reading of their release notes. I think it's worthwhile to figure this out first. Otherwise, there is a risk that any testing work turns out to be a wasted effort. --Chris Nauroth On 5/14/15, 7:23 AM, "Alan Burlison" <alan.burli...@oracle.com> wrote: >On 13/05/2015 17:13, Chris Nauroth wrote: > >> It was important to complete this upgrade before Hadoop 2.x came out of >> beta. After that, we committed to a policy of backwards-compatibility >> within the 2.x release line. I can't find a statement about whether or >> not Protocol Buffers 2.6.1 is backwards-compatible with 2.5.0 (both at >> compile time and on the wire). Do you know the answer? If it's >> backwards-incompatible, then we wouldn't be able to do this upgrade >>within >> Hadoop 2.x, though we could consider it for 3.x (trunk). > >I'm not sure about the wire format, what's the best way of checking for >wire format issues? > >http://upstream-tracker.org/versions/protobuf.html suggests there are >are some source-level issues which will require investigation. > >> In general, we upgrade dependencies when a new release offers a >>compelling >> benefit, not solely to keep up with the latest. In the case of 2.5.0, >> there was a performance benefit. Looking at the release notes for 2.6.0 >> and 2.6.1, I don't see anything particularly compelling. (That's just >>my >> opinion though, and others might disagree.) > >I think bundling or forking is the only practical option. I was looking >to see if we could provide ProtocolBuffers as an installable option on >our platform, if it's a version-compatibility nightmare as you say, >that's going to be difficult as we really don't want to have to provide >multiple versions. > >> BTW, if anyone is curious, it's possible to try a custom build right now >> linked against 2.6.1. You'd pass -Dprotobuf.version=2.6.1 and >> -Dprotoc.path=<path to protoc 2.6.1 binary> when you run the mvn >>command. > >Once I have fixed all the other source portability issues I'll circle >back around and take a look at this. > >-- >Alan Burlison >--
