Hi Eric, thanks for the investigation. - For YARN-6616, for branch-2.8 and below, it was only committed to 2.7.8/2.8.6 which have not been released (as I understand). Perhaps we can revert YARN-6616 from branch-2.7 and branch-2.8. - For YARN-6050, there's a bit here: https://developers.google.com/protocol-buffers/docs/proto that says "optional is compatible with repeated", so I think we should be OK there. - For YARN-7813, it's in 2.8.4 so it seems upgrading from 2.8.4 or 2.8.5 to a 2.9+ version will be an issue. One option could be to move the intraQueuePreemptionDisabled field from id 12 to id 13 in branch-2.8, then users would upgrade from 2.8.4/2.8.5 to 2.8.6 (someone would have to release this), then upgrade from 2.8.6 to 2.9+.
Jonathan Hung On Tue, Sep 24, 2019 at 9:23 AM Eric Badger <ebad...@verizonmedia.com.invalid> wrote: > We (Verizon Media) are currently moving towards upgrading our clusters from > our internal fork of branch-2.8 to an internal fork of branch-2. During > this process, we have found multiple incompatible changes in protobufs > between branch-2.8 and branch-2. These incompatibilities were all > introduced between branch-2.8 and branch-2.9. I did a git diff over all > .proto files across the branch-2.8 and branch-2.9 and found 3 instances of > incompatibilities from 3 separate commits. All of the incompatibilities are > in yarn_protos.proto > > > I would like to discuss how to fix these incompatible changes. Otherwise, > rolling upgrades will not be supported between branch-2.8 (or below) and > branch-2.9 (or beyond). We could revert the incompatible changes, but then > the new releases would be incompatible with the releases that have these > incompatible changes. If we do nothing, then rolling upgrades won't work > between 2.8- and 2.9+. > > > Thanks, > > > Eric > > > ------------------------------------------------------------------- > > > git diff branch-2.8..branch-2.9 $(find . -name '*\.proto') > > > https://issues.apache.org/jira/browse/YARN-6616 > > - Trunk patch (applied through branch-2.9) differs from branch-2.8 patch > > @@ -211,7 +245,20 @@ message ApplicationReportProto { > > optional PriorityProto priority = 23; > > optional string appNodeLabelExpression = 24; > > optional string amNodeLabelExpression = 25; > > - optional int64 submitTime = 26; > > + repeated AppTimeoutsMapProto appTimeouts = 26; > > + optional int64 launchTime = 27; > > + optional int64 submitTime = 28; > > > https://issues.apache.org/jira/browse/YARN-6050 > > - Trunk and branch-2 patches both change the protobuf type in the same > way. > > @@ -356,7 +416,22 @@ message ApplicationSubmissionContextProto { > > optional LogAggregationContextProto log_aggregation_context = 14; > > optional ReservationIdProto reservation_id = 15; > > optional string node_label_expression = 16; > > - optional ResourceRequestProto am_container_resource_request = 17; > > + repeated ResourceRequestProto am_container_resource_request = 17; > > + repeated ApplicationTimeoutMapProto application_timeouts = 18; > > > https://issues.apache.org/jira/browse/YARN-7813 > > - Trunk (applied through branch-3.1) and branch-3.0 (applied through > branch-2.9) patches differ from branch-2.8 patch > > @@ -425,7 +501,21 @@ message QueueInfoProto { > > optional string defaultNodeLabelExpression = 9; > > optional QueueStatisticsProto queueStatistics = 10; > > optional bool preemptionDisabled = 11; > > - optional bool intraQueuePreemptionDisabled = 12; > > + repeated QueueConfigurationsMapProto queueConfigurationsMap = 12; > > + optional bool intraQueuePreemptionDisabled = 13; >
