Thanks for confirming! I've just merged the PR, so this change should be in 1.46.
On Thu, Mar 17, 2022 at 3:21 PM František Bořánek <fbora...@gmail.com> wrote: > > Works like a charm. All current channels are included in a first message > during reconnection. > > Thanks very much. > > Here is from debug log: > > I0317 23:08:14.625789077 49 xds_client.cc:1287] [xds_client > 0x7f89b40352a0] xds server xds-control-plane:10000: ADS call status > received (chand=0x7f89b4042270, ads_calld=0x7f89f00fb600, > call=0x7f89d0022e70): status=1, details='Received RST_STREAM with error > code 8', error='"OK"' > I0317 23:08:14.625866170 49 xds_client.cc:685] [xds_client > 0x7f89b40352a0] xds server xds-control-plane:10000: start new call from > retryable call 0x7f89b4189aa0 > I0317 23:08:14.625891228 49 xds_client.cc:938] [xds_client > 0x7f89b40352a0] xds server xds-control-plane:10000: starting ADS call > (calld: 0x7f89b8095930, call: 0x7f89b81801f0) > D0317 23:08:14.626050477 49 xds_api.cc:257] [xds_client > 0x7f89b40352a0] constructed ADS request: version_info: > "8cbd8378-4243-4cc7-9a2f-1f075055a469" > node: { > cluster: "default-cluster" > user_agent_name: "gRPC C-core linux" > user_agent_version: "C-core 22.0.0" > client_features: "envoy.lb.does_not_support_overprovisioning" > 5: "gRPC C-core linux 22.0.0" > } > resource_names: "server-1" > resource_names: "server-2" > resource_names: "server-3" > resource_names: "server-4" > resource_names: "server-5" > resource_names: "server-6" > resource_names: "server-7" > type_url: "type.googleapis.com/envoy.api.v2.Listener" > > I0317 23:08:14.626073893 49 xds_client.cc:1055] [xds_client > 0x7f89b40352a0] xds server xds-control-plane:10000: sending ADS request: > type=envoy.config.listener.v3.Listener > version=8cbd8378-4243-4cc7-9a2f-1f075055a469 nonce= error="OK" > > > > > 17. 3. 2022 v 21:56, Mark D. Roth <r...@google.com>: > > Thanks for reporting this! I think I see the problem, and it should be > fixed by https://github.com/grpc/grpc/pull/29144. > > I don't see a trivial way to write a test for it, since the current > behavior is not broken, just slightly sub-optimal. But feel free to try > out the patch and let me know if it fixes the problem. > > On Thu, Mar 17, 2022 at 1:09 PM František Bořánek <fbora...@gmail.com> > wrote: > >> It's not exactly the same version. This one is gRPC 1.43.2, but behaviour >> do not vary form 1.44.0. >> I attached also initial start up, in case you want to compare. >> Reconnection of xds contains file goaway.from.xds.cpp.grpc.log. >> The lifetime of channels are until shutdown of the application. >> >> >> >> 17. 3. 2022 v 16:52, Mark D. Roth <r...@google.com>: >> >> That's very interesting. Can you run with the following environment >> variables and send me the log? >> >> GRPC_VERBOSITY=DEBUG GRPC_TRACE=xds_client,xds_resolver >> >> On Wed, Mar 16, 2022 at 10:53 PM František Bořánek <fbora...@gmail.com> >> wrote: >> >>> C++, the latest 1.44.0 release. >>> >>> 16. 3. 2022 v 21:32, Mark D. Roth <r...@google.com>: >>> >>> >>> It is valid behavior, but I agree that it's a little sub-optimal for the >>> client to send two messages when it reestablishes the stream. What >>> language are you using gRPC in? >>> >>> On Wed, Mar 16, 2022 at 1:19 PM František Bořánek <fbora...@gmail.com> >>> wrote: >>> >>>> It makes sense. >>>> >>>> However, in this case, the initial connection was already made and this >>>> tcpdump samples recovering of ADS stream. And as you write: >>>> >>>> > Note that once your application receives all 7 resources from the xDS >>>> server, if the ADS stream is broken and the client reestablishes it at that >>>> point, it should immediately subscribe to all 7 resources in the initial >>>> message on the new stream. >>>> >>>> It behaves as I described. First 1 resource and then 7 resources. Note >>>> that the messages have version_info. Brand new channels don't have >>>> version_info. >>>> >>>> But as you pointed, the channels can be created anytime, thus it's >>>> valid bahaviur. >>>> >>>> Thanks for replying. >>>> >>>> F. >>>> >>>> 16. 3. 2022 v 19:49, Mark D. Roth <r...@google.com>: >>>> >>>> The gRPC client does not know a priori what set of resource names are >>>> available on the xDS server, and even if it did, it would not request all >>>> of them proactively, because it may not actually need all of them. >>>> Instead, each time a gRPC channel is created with an "xds:" target URI, >>>> that tells gRPC which LDS resource to request, and it requests the >>>> resources as they are needed. Because the application can create new >>>> channels at any time, the set of resources requested on the ADS stream can >>>> change at any time. >>>> >>>> If your application immediately creates 7 channels when it starts up, >>>> then the pattern you're seeing is expected. When the first channel >>>> (presumably for "xds:server-1") starts connecting, it will immediately >>>> subscribe to the corresponding LDS resource ("server-1"). While it's >>>> waiting for that message to be sent, it notices that the other 6 channels >>>> have started up, so it realizes that it has to subscribe to the other 6 >>>> resources. By the time the initial message is sent and it's able to send a >>>> second message, it knows about all 7 subscriptions, so the second message >>>> contains all 7 of them. >>>> >>>> Note that once your application receives all 7 resources from the xDS >>>> server, if the ADS stream is broken and the client reestablishes it at that >>>> point, it should immediately subscribe to all 7 resources in the initial >>>> message on the new stream. That is because it already knows about all of >>>> the subscriptions at the point at which it sends the initial message on the >>>> new stream. >>>> >>>> This communication is fully legal and should work fine with any >>>> xDS-compliant server. As described in this section of the xDS spec >>>> <https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#ack-nack-and-resource-type-instance-version>, >>>> the intent of the version field in the DiscoveryRequest is to provide the >>>> server with a way to know when it sends a version that the client considers >>>> invalid. But if the client sends a second DiscoveryRequest with the same >>>> version and a different list of resource names, that indicates that it is >>>> changing what resources it is subscribing to, and the server cannot ignore >>>> that just because the version is the same as one it has previously seen. >>>> >>>> I hope this information is helpful. >>>> >>>> On Mon, Mar 14, 2022 at 5:41 AM František Bořánek <fbora...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> I don't know if such behaviour is a bug. >>>>> >>>>> I have server communicates with 7 services configured via XDS and >>>>> after the xds-client got GO AWAY packet from XDS server 6 channels cannot >>>>> connect due to error and only one is ok. I did a tcpdump to see a chat >>>>> between our implementation of XDS and C++gRPC and cannot deside if C++gRPC >>>>> behave correct. >>>>> >>>>> It relates to >>>>> https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md#initial-request-on-the-ads-stream >>>>> > The first request on the ADS stream will include a Node >>>>> message that identifies the client. As mentioned above, most of the fields >>>>> for the Node message will be read from the bootstrap file. >>>>> >>>>> I would expect that after reconnect the xds client sends to ADS stream >>>>> 4 requests (LDS, CDS, EDS, RDS) however it sends 5 requests (LDS, LDS, >>>>> CDS, >>>>> EDS, RDS). On our side of xds server we are confused by these 2 LDS >>>>> requests. >>>>> >>>>> 1. >>>>> (PROTOBUF) envoy.api.v2.DiscoveryRequest >>>>> { >>>>> version_info: "ba0255a2-07b6-45ed-83b4-dad0cbf96b1f" >>>>> node: { >>>>> cluster: "default-cluster" >>>>> user_agent_name: "gRPC C-core linux" >>>>> user_agent_version: "C-core 22.0.0" >>>>> client_features: "envoy.lb.does_not_support_overprovisioning" >>>>> build_version: "gRPC C-core linux 22.0.0" >>>>> } >>>>> resource_names: "server-1" >>>>> type_url: "type.googleapis.com/envoy.api.v2.Listener" >>>>> } >>>>> >>>>> 2. >>>>> (PROTOBUF) envoy.api.v2.DiscoveryRequest >>>>> { >>>>> version_info: "ba0255a2-07b6-45ed-83b4-dad0cbf96b1f" >>>>> resource_names: "server-1" >>>>> resource_names: "server-2" >>>>> resource_names: "server-3" >>>>> resource_names: "server-4" >>>>> resource_names: "server-5" >>>>> resource_names: "server-6" >>>>> resource_names: "server-7" >>>>> type_url: "type.googleapis.com/envoy.api.v2.Listener" >>>>> } >>>>> >>>>> - Shouldn't it be only one LDS request? >>>>> - Shouldn't it the initial request contains all configured resources? >>>>> >>>>> Root of our problem is that we responds only on first request since it >>>>> have the same version. I know how to fix it on our end, but it is >>>>> behaviour >>>>> of gRPC xds-client correct? >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "grpc.io" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to grpc-io+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/grpc-io/e2a5befd-5943-4b40-afc4-4864938b9b88n%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/grpc-io/e2a5befd-5943-4b40-afc4-4864938b9b88n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> >>>> >>>> -- >>>> Mark D. Roth <r...@google.com> >>>> Software Engineer >>>> Google, Inc. >>>> >>>> >>>> >>> >>> -- >>> Mark D. Roth <r...@google.com> >>> Software Engineer >>> Google, Inc. >>> >>> >> >> -- >> Mark D. Roth <r...@google.com> >> Software Engineer >> Google, Inc. >> >> >> > > -- > Mark D. Roth <r...@google.com> > Software Engineer > Google, Inc. > > > -- Mark D. Roth <r...@google.com> Software Engineer Google, Inc. -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAJgPXp6%3D-dmPt4ZmhZX-p8w5CJVO7xB6BQFmKGps-qhiK0o3hQ%40mail.gmail.com.