Works like a charm. All current channels are included in a first message during
reconnection.
Thanks very much.
Here is from debug log:
I0317 23:08:14.625789077 49 xds_client.cc:1287] [xds_client
0x7f89b40352a0] xds server xds-control-plane:10000: ADS call status received
(chand=0x7f89b4042270, ads_calld=0x7f89f00fb600, call=0x7f89d0022e70):
status=1, details='Received RST_STREAM with error code 8', error='"OK"'
I0317 23:08:14.625866170 49 xds_client.cc:685] [xds_client
0x7f89b40352a0] xds server xds-control-plane:10000: start new call from
retryable call 0x7f89b4189aa0
I0317 23:08:14.625891228 49 xds_client.cc:938] [xds_client
0x7f89b40352a0] xds server xds-control-plane:10000: starting ADS call (calld:
0x7f89b8095930, call: 0x7f89b81801f0)
D0317 23:08:14.626050477 49 xds_api.cc:257] [xds_client
0x7f89b40352a0] constructed ADS request: version_info:
"8cbd8378-4243-4cc7-9a2f-1f075055a469"
node: {
cluster: "default-cluster"
user_agent_name: "gRPC C-core linux"
user_agent_version: "C-core 22.0.0"
client_features: "envoy.lb.does_not_support_overprovisioning"
5: "gRPC C-core linux 22.0.0"
}
resource_names: "server-1"
resource_names: "server-2"
resource_names: "server-3"
resource_names: "server-4"
resource_names: "server-5"
resource_names: "server-6"
resource_names: "server-7"
type_url: "type.googleapis.com/envoy.api.v2.Listener"
I0317 23:08:14.626073893 49 xds_client.cc:1055] [xds_client
0x7f89b40352a0] xds server xds-control-plane:10000: sending ADS request:
type=envoy.config.listener.v3.Listener
version=8cbd8378-4243-4cc7-9a2f-1f075055a469 nonce= error="OK"
> 17. 3. 2022 v 21:56, Mark D. Roth <r...@google.com>:
>
> Thanks for reporting this! I think I see the problem, and it should be fixed
> by https://github.com/grpc/grpc/pull/29144
> <https://github.com/grpc/grpc/pull/29144>.
>
> I don't see a trivial way to write a test for it, since the current behavior
> is not broken, just slightly sub-optimal. But feel free to try out the patch
> and let me know if it fixes the problem.
>
> On Thu, Mar 17, 2022 at 1:09 PM František Bořánek <fbora...@gmail.com
> <mailto:fbora...@gmail.com>> wrote:
> It's not exactly the same version. This one is gRPC 1.43.2, but behaviour do
> not vary form 1.44.0.
> I attached also initial start up, in case you want to compare. Reconnection
> of xds contains file goaway.from.xds.cpp.grpc.log.
> The lifetime of channels are until shutdown of the application.
>
>
>
>> 17. 3. 2022 v 16:52, Mark D. Roth <r...@google.com <mailto:r...@google.com>>:
>>
>> That's very interesting. Can you run with the following environment
>> variables and send me the log?
>>
>> GRPC_VERBOSITY=DEBUG GRPC_TRACE=xds_client,xds_resolver
>>
>> On Wed, Mar 16, 2022 at 10:53 PM František Bořánek <fbora...@gmail.com
>> <mailto:fbora...@gmail.com>> wrote:
>> C++, the latest 1.44.0 release.
>>
>>> 16. 3. 2022 v 21:32, Mark D. Roth <r...@google.com
>>> <mailto:r...@google.com>>:
>>>
>>>
>>> It is valid behavior, but I agree that it's a little sub-optimal for the
>>> client to send two messages when it reestablishes the stream. What
>>> language are you using gRPC in?
>>>
>>> On Wed, Mar 16, 2022 at 1:19 PM František Bořánek <fbora...@gmail.com
>>> <mailto:fbora...@gmail.com>> wrote:
>>> It makes sense.
>>>
>>> However, in this case, the initial connection was already made and this
>>> tcpdump samples recovering of ADS stream. And as you write:
>>>
>>> > Note that once your application receives all 7 resources from the xDS
>>> > server, if the ADS stream is broken and the client reestablishes it at
>>> > that point, it should immediately subscribe to all 7 resources in the
>>> > initial message on the new stream.
>>>
>>> It behaves as I described. First 1 resource and then 7 resources. Note that
>>> the messages have version_info. Brand new channels don't have version_info.
>>>
>>> But as you pointed, the channels can be created anytime, thus it's valid
>>> bahaviur.
>>>
>>> Thanks for replying.
>>>
>>> F.
>>>
>>>> 16. 3. 2022 v 19:49, Mark D. Roth <r...@google.com
>>>> <mailto:r...@google.com>>:
>>>>
>>>> The gRPC client does not know a priori what set of resource names are
>>>> available on the xDS server, and even if it did, it would not request all
>>>> of them proactively, because it may not actually need all of them.
>>>> Instead, each time a gRPC channel is created with an "xds:" target URI,
>>>> that tells gRPC which LDS resource to request, and it requests the
>>>> resources as they are needed. Because the application can create new
>>>> channels at any time, the set of resources requested on the ADS stream can
>>>> change at any time.
>>>>
>>>> If your application immediately creates 7 channels when it starts up, then
>>>> the pattern you're seeing is expected. When the first channel (presumably
>>>> for "xds:server-1") starts connecting, it will immediately subscribe to
>>>> the corresponding LDS resource ("server-1"). While it's waiting for that
>>>> message to be sent, it notices that the other 6 channels have started up,
>>>> so it realizes that it has to subscribe to the other 6 resources. By the
>>>> time the initial message is sent and it's able to send a second message,
>>>> it knows about all 7 subscriptions, so the second message contains all 7
>>>> of them.
>>>>
>>>> Note that once your application receives all 7 resources from the xDS
>>>> server, if the ADS stream is broken and the client reestablishes it at
>>>> that point, it should immediately subscribe to all 7 resources in the
>>>> initial message on the new stream. That is because it already knows about
>>>> all of the subscriptions at the point at which it sends the initial
>>>> message on the new stream.
>>>>
>>>> This communication is fully legal and should work fine with any
>>>> xDS-compliant server. As described in this section of the xDS spec
>>>> <https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol#ack-nack-and-resource-type-instance-version>,
>>>> the intent of the version field in the DiscoveryRequest is to provide the
>>>> server with a way to know when it sends a version that the client
>>>> considers invalid. But if the client sends a second DiscoveryRequest with
>>>> the same version and a different list of resource names, that indicates
>>>> that it is changing what resources it is subscribing to, and the server
>>>> cannot ignore that just because the version is the same as one it has
>>>> previously seen.
>>>>
>>>> I hope this information is helpful.
>>>>
>>>> On Mon, Mar 14, 2022 at 5:41 AM František Bořánek <fbora...@gmail.com
>>>> <mailto:fbora...@gmail.com>> wrote:
>>>> Hi,
>>>> I don't know if such behaviour is a bug.
>>>>
>>>> I have server communicates with 7 services configured via XDS and after
>>>> the xds-client got GO AWAY packet from XDS server 6 channels cannot
>>>> connect due to error and only one is ok. I did a tcpdump to see a chat
>>>> between our implementation of XDS and C++gRPC and cannot deside if C++gRPC
>>>> behave correct.
>>>>
>>>> It relates to
>>>> https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md#initial-request-on-the-ads-stream
>>>>
>>>> <https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md#initial-request-on-the-ads-stream>
>>>> > The first request on the ADS stream will include a Node message
>>>> that identifies the client. As mentioned above, most of the fields for the
>>>> Node message will be read from the bootstrap file.
>>>>
>>>> I would expect that after reconnect the xds client sends to ADS stream 4
>>>> requests (LDS, CDS, EDS, RDS) however it sends 5 requests (LDS, LDS, CDS,
>>>> EDS, RDS). On our side of xds server we are confused by these 2 LDS
>>>> requests.
>>>>
>>>> 1.
>>>> (PROTOBUF) envoy.api.v2.DiscoveryRequest
>>>> {
>>>> version_info: "ba0255a2-07b6-45ed-83b4-dad0cbf96b1f"
>>>> node: {
>>>> cluster: "default-cluster"
>>>> user_agent_name: "gRPC C-core linux"
>>>> user_agent_version: "C-core 22.0.0"
>>>> client_features: "envoy.lb.does_not_support_overprovisioning"
>>>> build_version: "gRPC C-core linux 22.0.0"
>>>> }
>>>> resource_names: "server-1"
>>>> type_url: "type.googleapis.com/envoy.api.v2.Listener
>>>> <http://type.googleapis.com/envoy.api.v2.Listener>"
>>>> }
>>>>
>>>> 2.
>>>> (PROTOBUF) envoy.api.v2.DiscoveryRequest
>>>> {
>>>> version_info: "ba0255a2-07b6-45ed-83b4-dad0cbf96b1f"
>>>> resource_names: "server-1"
>>>> resource_names: "server-2"
>>>> resource_names: "server-3"
>>>> resource_names: "server-4"
>>>> resource_names: "server-5"
>>>> resource_names: "server-6"
>>>> resource_names: "server-7"
>>>> type_url: "type.googleapis.com/envoy.api.v2.Listener
>>>> <http://type.googleapis.com/envoy.api.v2.Listener>"
>>>> }
>>>>
>>>> - Shouldn't it be only one LDS request?
>>>> - Shouldn't it the initial request contains all configured resources?
>>>>
>>>> Root of our problem is that we responds only on first request since it
>>>> have the same version. I know how to fix it on our end, but it is
>>>> behaviour of gRPC xds-client correct?
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "grpc.io <http://grpc.io/>" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to grpc-io+unsubscr...@googlegroups.com
>>>> <mailto:grpc-io+unsubscr...@googlegroups.com>.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/grpc-io/e2a5befd-5943-4b40-afc4-4864938b9b88n%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/grpc-io/e2a5befd-5943-4b40-afc4-4864938b9b88n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>>>>
>>>>
>>>> --
>>>> Mark D. Roth <r...@google.com <mailto:r...@google.com>>
>>>> Software Engineer
>>>> Google, Inc.
>>>
>>>
>>>
>>> --
>>> Mark D. Roth <r...@google.com <mailto:r...@google.com>>
>>> Software Engineer
>>> Google, Inc.
>>
>>
>> --
>> Mark D. Roth <r...@google.com <mailto:r...@google.com>>
>> Software Engineer
>> Google, Inc.
>
>
>
> --
> Mark D. Roth <r...@google.com <mailto:r...@google.com>>
> Software Engineer
> Google, Inc.
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to grpc-io+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/0323D426-3B27-4F8D-B4F9-DF76DEBE39ED%40gmail.com.
