Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[tom petch]

From: Rob Wilton (rwilton) 
Sent: 23 September 2022 12:53

Hi Tom,

Perhaps you can suggest some text/clarifications and the authors could consider 
it as part of the IETF last-call?


Rob

Consistency is my biggest concern with -10.  Search on 'underlay' and I see 
underlay network
underlay networks
underlay link
underlay technology
underlay-tunnel
underlay tunnels
underlay transport type
underlay network instances
underlay network links 
underlay network termination  points 

which leaves me uncertain, confused even and this I think should be addressed.  
I see similar variation in the use of 'overlay, 'VPN', 'service'.  

The difference between 'network' and 'networks' may be slight but I find it 
material.  The I-D talks of Layer 3 and Layer 2 in a stack.  My sense of the 
I-D is a focus on a L2VPN over a Layer 2 network.  Layer 3 and L3VPN get a 
mention but  L3VPN will have a Layer 3 underlay network which will have a Layer 
2 underlay network but in places, the use of 'network' singular suggests that 
this is not what the model is about, that the model only countenances a single 
underlay network to a service network.  One of the changes in the I-D was the 
augment of 'link' becoming a list, that there might be more than one link 
involved, and I have the same uncertainty about 'network'. Is it just the one 
underlay and service or more than one and if more than one, what are the 
relationships?  I do not know .  It is the sort of uncertainty that some Genart 
reviewers are good at spotting so I will wait to see what else Last Call brings 
up.

Tom Petch
Thanks,
Rob


> -Original Message-
> From: tom petch 
> Sent: 23 September 2022 12:20
> To: Rob Wilton (rwilton) ; 'Wubo (lana)'
> ; draft-ietf-opsawg-yang-vpn-service-
> pm@ietf.org; adr...@olddog.co.uk
> Cc: opsawg@ietf.org
> Subject: Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-
> 09
>
> From: Adrian Farrel 
> Sent: 16 September 2022 10:33
>
> Hi Tom, all,
>
> I think my review as Shepherd ran into the same concern. And it is one of my
> long-standing gripes that "we" (the IETF) repeatedly confuse VPN as a service
> with the means and mechanisms to realise the VPN within the network. Of
> course, as network engineers, it is understandable why we make that
> mistake, but it is also harmful to the way we talk about the customers' view
> of VPNs.
>
> Now, in discussing this document with the authors, I wanted to distinguish
> between the performance measurement that the customer can perform
> (which is strictly edge-to-edge because the customer cannot see what is
> happening within the network), and the measurements that the provider can
> perform that can be far more analytic about the resources and routes/paths
> within the network. My feeling was that the authors completely got this
> distinction, but that they wanted to look at the performance monitoring from
> the provider's perspective with two viewpoints: what can they measure
> about how their network is performing, and what can they measure that will
> match what the customer might measure. Of course, the provider wants to
> know the latter before the customer notices and complains, but the provider
> also wants to be able to link the edge-to-edge measurements back to the
> more detailed measurements from within the network to determine causes.
>
> It is possible that I have expressed that differently from the way the
> document describes it, and it also possible that I have misrepresented the
> authors and the working group. But that was my take-away.
>
> 
>
> Adrian
>
> As I expect you will have seen, I took a look at -10 and still get confused.  
> It
> seems that several different  terms get used and I am left uncertain as to
> whether or not it is just two concepts,, 'underlay networks and overlay VPN
> services' to quote the Abstract, or if there is more involved.
>
> From your discussion with the authors I think just two, but I do not find the
> body of the I-D clear on that.
> Tom Petch
>
> Cheers,
> Adrian
>
> -Original Message-
> From: OPSAWG  On Behalf Of tom petch
> Sent: 15 September 2022 11:37
>
> From: OPSAWG  on behalf of Rob Wilton
> (rwilton) 
> Sent: 15 September 2022 09:09
>
> Hi Bo,
>
> Looks good.
>
> Let me know when you have an updated version of the draft posted and I
> will kick off the IETF LC.
>
> Thanks for the updates and for taking my comments onboard.
>
> 
> I have been following this thread with a sense of deja vu having made similar
> comments, much on s.4.2 , back in May.  Except, I do not think I ever hit
> 'send'.  I was trying to make clear comments that were not confused but
> found the I-D so confusing that I kept on changing my comments to try and
> make them 

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Rob Wilton (rwilton)]

Hi Tom,

Perhaps you can suggest some text/clarifications and the authors could consider 
it as part of the IETF last-call?

Thanks,
Rob


> -Original Message-
> From: tom petch 
> Sent: 23 September 2022 12:20
> To: Rob Wilton (rwilton) ; 'Wubo (lana)'
> ; draft-ietf-opsawg-yang-vpn-service-
> pm@ietf.org; adr...@olddog.co.uk
> Cc: opsawg@ietf.org
> Subject: Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-
> 09
> 
> From: Adrian Farrel 
> Sent: 16 September 2022 10:33
> 
> Hi Tom, all,
> 
> I think my review as Shepherd ran into the same concern. And it is one of my
> long-standing gripes that "we" (the IETF) repeatedly confuse VPN as a service
> with the means and mechanisms to realise the VPN within the network. Of
> course, as network engineers, it is understandable why we make that
> mistake, but it is also harmful to the way we talk about the customers' view
> of VPNs.
> 
> Now, in discussing this document with the authors, I wanted to distinguish
> between the performance measurement that the customer can perform
> (which is strictly edge-to-edge because the customer cannot see what is
> happening within the network), and the measurements that the provider can
> perform that can be far more analytic about the resources and routes/paths
> within the network. My feeling was that the authors completely got this
> distinction, but that they wanted to look at the performance monitoring from
> the provider's perspective with two viewpoints: what can they measure
> about how their network is performing, and what can they measure that will
> match what the customer might measure. Of course, the provider wants to
> know the latter before the customer notices and complains, but the provider
> also wants to be able to link the edge-to-edge measurements back to the
> more detailed measurements from within the network to determine causes.
> 
> It is possible that I have expressed that differently from the way the
> document describes it, and it also possible that I have misrepresented the
> authors and the working group. But that was my take-away.
> 
> 
> 
> Adrian
> 
> As I expect you will have seen, I took a look at -10 and still get confused.  
> It
> seems that several different  terms get used and I am left uncertain as to
> whether or not it is just two concepts,, 'underlay networks and overlay VPN
> services' to quote the Abstract, or if there is more involved.
> 
> From your discussion with the authors I think just two, but I do not find the
> body of the I-D clear on that.
> Tom Petch
> 
> Cheers,
> Adrian
> 
> -Original Message-
> From: OPSAWG  On Behalf Of tom petch
> Sent: 15 September 2022 11:37
> 
> From: OPSAWG  on behalf of Rob Wilton
> (rwilton) 
> Sent: 15 September 2022 09:09
> 
> Hi Bo,
> 
> Looks good.
> 
> Let me know when you have an updated version of the draft posted and I
> will kick off the IETF LC.
> 
> Thanks for the updates and for taking my comments onboard.
> 
> 
> I have been following this thread with a sense of deja vu having made similar
> comments, much on s.4.2 , back in May.  Except, I do not think I ever hit
> 'send'.  I was trying to make clear comments that were not confused but
> found the I-D so confusing that I kept on changing my comments to try and
> make them clear and never finished.
> 
> My comments were that the document contradicted the Abstract, that the I-
> D was mostly about VPN services and not about network level.  I concluded
> that this I-D was really two separate pieces of work, headed for two separate
> RFC, banged together because they had some groupings in common, and I
> think that much of the discussion in this thread has revolved around that
> issue.  (It is a bit like YANG modules with masses of groupings which save the
> author repeating a few lines of YANG while making it harder for anyone else
> to follow, except more so).
> 
> So, I shall try to forget what I have learnt from this thread and read the
> revised I-D to see if I find it any clearer but will probably end up with the
> same conclusion, this is two separate RFC.
> 
> Tom Petch.
> 
> Regards,
> Rob
> 
> 
> From: Wubo (lana) 
> Sent: 15 September 2022 03:17
> 
> Hi Rob,
> 
> Thank you for the review and helpful comments.
> 
> I copied your last comment here, since this is the last point to be discussed.
> 
> RW3:
> Based on your additional information, then I think that saying that is does
> not allow the gathering of performance data simultaneously is somewhat
> confusing.  E.g., you could make a get request that spanned over multiple
> network list entries, or similar for a subscription.
> 
>

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[tom petch]

From: Adrian Farrel 
Sent: 16 September 2022 10:33

Hi Tom, all,

I think my review as Shepherd ran into the same concern. And it is one of my 
long-standing gripes that "we" (the IETF) repeatedly confuse VPN as a service 
with the means and mechanisms to realise the VPN within the network. Of course, 
as network engineers, it is understandable why we make that mistake, but it is 
also harmful to the way we talk about the customers' view of VPNs.

Now, in discussing this document with the authors, I wanted to distinguish 
between the performance measurement that the customer can perform (which is 
strictly edge-to-edge because the customer cannot see what is happening within 
the network), and the measurements that the provider can perform that can be 
far more analytic about the resources and routes/paths within the network. My 
feeling was that the authors completely got this distinction, but that they 
wanted to look at the performance monitoring from the provider's perspective 
with two viewpoints: what can they measure about how their network is 
performing, and what can they measure that will match what the customer might 
measure. Of course, the provider wants to know the latter before the customer 
notices and complains, but the provider also wants to be able to link the 
edge-to-edge measurements back to the more detailed measurements from within 
the network to determine causes.

It is possible that I have expressed that differently from the way the document 
describes it, and it also possible that I have misrepresented the authors and 
the working group. But that was my take-away.



Adrian

As I expect you will have seen, I took a look at -10 and still get confused.  
It seems that several different  terms get used and I am left uncertain as to 
whether or not it is just two concepts,, 'underlay networks and overlay VPN 
services' to quote the Abstract, or if there is more involved.

From your discussion with the authors I think just two, but I do not find the 
body of the I-D clear on that.
Tom Petch
 
Cheers,
Adrian

-Original Message-
From: OPSAWG  On Behalf Of tom petch
Sent: 15 September 2022 11:37

From: OPSAWG  on behalf of Rob Wilton (rwilton) 

Sent: 15 September 2022 09:09

Hi Bo,

Looks good.

Let me know when you have an updated version of the draft posted and I will 
kick off the IETF LC.

Thanks for the updates and for taking my comments onboard.


I have been following this thread with a sense of deja vu having made similar 
comments, much on s.4.2 , back in May.  Except, I do not think I ever hit 
'send'.  I was trying to make clear comments that were not confused but found 
the I-D so confusing that I kept on changing my comments to try and make them 
clear and never finished.

My comments were that the document contradicted the Abstract, that the I-D was 
mostly about VPN services and not about network level.  I concluded that this 
I-D was really two separate pieces of work, headed for two separate RFC, banged 
together because they had some groupings in common, and I think that much of 
the discussion in this thread has revolved around that issue.  (It is a bit 
like YANG modules with masses of groupings which save the author repeating a 
few lines of YANG while making it harder for anyone else to follow, except more 
so).

So, I shall try to forget what I have learnt from this thread and read the 
revised I-D to see if I find it any clearer but will probably end up with the 
same conclusion, this is two separate RFC.

Tom Petch.

Regards,
Rob


From: Wubo (lana) 
Sent: 15 September 2022 03:17

Hi Rob,

Thank you for the review and helpful comments.

I copied your last comment here, since this is the last point to be discussed.

RW3:
Based on your additional information, then I think that saying that is does not 
allow the gathering of performance data simultaneously is somewhat confusing.  
E.g., you could make a get request that spanned over multiple network list 
entries, or similar for a subscription.

I think that probably nothing extra needs to be said at all.  But if you do 
want to add text here then I suggest that it clarifies that networks and VPNs 
would be separate entries in the network list, and the underlying network would 
not have the “service” container set, whereas the VPN network entries would.

Bo4: Thanks for the suggestion. How about the changes:

==

4.2.  Network Level

The model can be used for performance monitoring both for the network and the 
VPN services. However, the module does not allow to gather the performance 
monitoring data simultaneously for both cases. Concretely: The two would be 
separate entries in the network list. The differences are as follows:

* When the “service-type” presence container is absent, then it indicates
performance monitoring of the network itself.

* When the “service-type” presence container is present, then it indicates
performance monitoring of the VPN service specified by the “service-type”
leaf, 

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Wubo (lana)]

Hi Rob,



Thanks for your accurate summary and further review. Please see inline.



Best regards,

Bo



-Original Message-
From: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
Sent: Friday, September 16, 2022 8:39 PM
To: adr...@olddog.co.uk; 'tom petch' ; Wubo (lana) 
; draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
Cc: opsawg@ietf.org
Subject: RE: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



My interpretation of the draft was basically this:



(1) The YANG topology model (rfc8345) can model both an underlay network and 
overlaying services.

(2) The base YANG topology model is missing some generic attributes to identify 
that a topology represents a service (e.g., service-type, vpn-id, 
vpn-service-topology).  I don't think that these attributes necessarily have 
anything to with PM, but they were added here because they were needed.  E.g., 
arguably they could have been put into a separate YANG module, but it would 
perhaps be too small to be worthwhile).

(3) The performance monitoring data can largely be gathered either at the 
network layer or at the service layer and this is really distinguished by which 
entry in the topology list the PM data nodes are being returned for.



Authors, is my understanding correct and accurate?



[Bo Wu] Thanks for the accurate summary.

On the second point, we agree that these VPN attributes are not PM data, but we 
think these are context information, for example, SLA requirements are 
different between hubs and between hub and spokes.



For (2), that does raise a further question:  In section 4.3, the "role" leaf 
has been placed under pm-attributes.  But again, I wonder whether this "role" 
is really just a generic description of the service endpoint.  Hence, would it 
be better to name this "vpn-service-role" and augment it directly under 
/nw:networks/nw:network/nw:node?  Possibly, it could be made conditional on 
/nw:networks/nw:network/nw:network-types/service/service-type



[Bo Wu] Thank you for pointing this out. Yes, we can take this out the PM 
attributes. How about we reconstructed YANG as follows:

  augment /nw:networks/nw:network/nw:node:

+--rw node-type?   identityref

+--ro entry-summary

   +--ro ipv4-num

   |  +--ro maximum-routes?uint32

   |  +--ro total-active-routes?   uint32

   +--ro ipv6-num

   |  +--ro maximum-routes?uint32

   |  +--ro total-active-routes?   uint32

   +--ro mac-num

  +--ro mac-num-limit?  uint32

  +--ro total-active-mac-num?   uint32

  augment /nw:networks/nw:network/nw:node:

+--rw role?   Identityref



For the complete change, please find at:

Diff:   
https://www.ietf.org/rfcdiff?url2=draft-ietf-opsawg-yang-vpn-service-pm-10



Rob







> -Original Message-

> From: Adrian Farrel mailto:adr...@olddog.co.uk>>

> Sent: 16 September 2022 10:34

> To: 'tom petch' mailto:ie...@btconnect.com>>; Rob Wilton 
> (rwilton)

> mailto:rwil...@cisco.com>>; 'Wubo (lana)' 
> mailto:lana.w...@huawei.com>>; draft-ietf-

> opsawg-yang-vpn-service-pm@ietf.org<mailto:opsawg-yang-vpn-service-pm@ietf.org>

> Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>

> Subject: RE: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-

> pm-09

>

> Hi Tom, all,

>

> I think my review as Shepherd ran into the same concern. And it is one

> of my long-standing gripes that "we" (the IETF) repeatedly confuse VPN

> as a service with the means and mechanisms to realise the VPN within

> the network. Of course, as network engineers, it is understandable why

> we make that mistake, but it is also harmful to the way we talk about

> the customers' view of VPNs.

>

> Now, in discussing this document with the authors, I wanted to

> distinguish between the performance measurement that the customer can

> perform (which is strictly edge-to-edge because the customer cannot

> see what is happening within the network), and the measurements that

> the provider can perform that can be far more analytic about the

> resources and routes/paths within the network. My feeling was that the

> authors completely got this distinction, but that they wanted to look

> at the performance monitoring from the provider's perspective with two

> viewpoints: what can they measure about how their network is

> performing, and what can they measure that will match what the

> customer might measure. Of course, the provider wants to know the

> latter before the customer notices and complains, but the provider

> also wants to be able to link the edge-to-edge measurements back to

> the more detailed measurements from within the network to determine causes.

>

> It is possible that I have expressed that differently from the way the

&g

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Wubo (lana)]

Hi Adrian, all,



Many thanks for your shepherd. Please see inline.



Best regards,

Bo



-Original Message-
From: Adrian Farrel [mailto:adr...@olddog.co.uk]
Sent: Friday, September 16, 2022 5:34 PM
To: 'tom petch' ; 'Rob Wilton (rwilton)' 
; Wubo (lana) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
Cc: opsawg@ietf.org
Subject: RE: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



Hi Tom, all,



I think my review as Shepherd ran into the same concern. And it is one of my 
long-standing gripes that "we" (the IETF) repeatedly confuse VPN as a service 
with the means and mechanisms to realise the VPN within the network. Of course, 
as network engineers, it is understandable why we make that mistake, but it is 
also harmful to the way we talk about the customers' view of VPNs.



Now, in discussing this document with the authors, I wanted to distinguish 
between the performance measurement that the customer can perform (which is 
strictly edge-to-edge because the customer cannot see what is happening within 
the network), and the measurements that the provider can perform that can be 
far more analytic about the resources and routes/paths within the network. My 
feeling was that the authors completely got this distinction, but that they 
wanted to look at the performance monitoring from the provider's perspective 
with two viewpoints: what can they measure about how their network is 
performing, and what can they measure that will match what the customer might 
measure. Of course, the provider wants to know the latter before the customer 
notices and complains, but the provider also wants to be able to link the 
edge-to-edge measurements back to the more detailed measurements from within 
the network to determine causes.



[Bo Wu] Thanks for your good summary. This is the exact purpose of this model. 
To achieve this, we also consider the similar modeling method of RFC 8345, as 
both the underlay network and the overlay network share the same metrics and 
counters.



It is possible that I have expressed that differently from the way the document 
describes it, and it also possible that I have misrepresented the authors and 
the working group. But that was my take-away.



Cheers,

Adrian



-Original Message-

From: OPSAWG mailto:opsawg-boun...@ietf.org>> On 
Behalf Of tom petch

Sent: 15 September 2022 11:37

To: Rob Wilton (rwilton) 
mailto:rwilton=40cisco@dmarc.ietf.org>>;
 Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>

Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>

Subject: Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



From: OPSAWG mailto:opsawg-boun...@ietf.org>> on 
behalf of Rob Wilton (rwilton) 
mailto:rwilton=40cisco@dmarc.ietf.org>>

Sent: 15 September 2022 09:09



Hi Bo,



Looks good.



Let me know when you have an updated version of the draft posted and I will 
kick off the IETF LC.



Thanks for the updates and for taking my comments onboard.





I have been following this thread with a sense of deja vu having made similar 
comments, much on s.4.2 , back in May.  Except, I do not think I ever hit 
'send'.  I was trying to make clear comments that were not confused but found 
the I-D so confusing that I kept on changing my comments to try and make them 
clear and never finished.



My comments were that the document contradicted the Abstract, that the I-D was 
mostly about VPN services and not about network level.  I concluded that this 
I-D was really two separate pieces of work, headed for two separate RFC, banged 
together because they had some groupings in common, and I think that much of 
the discussion in this thread has revolved around that issue.  (It is a bit 
like YANG modules with masses of groupings which save the author repeating a 
few lines of YANG while making it harder for anyone else to follow, except more 
so).



So, I shall try to forget what I have learnt from this thread and read the 
revised I-D to see if I find it any clearer but will probably end up with the 
same conclusion, this is two separate RFC.



Tom Petch.



Regards,

Rob





From: Wubo (lana) mailto:lana.w...@huawei.com>>

Sent: 15 September 2022 03:17

To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm....@ietf.org>

Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>

Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



Hi Rob,



Thank you for the review and helpful comments.



I copied your last comment here, since this is the last point to be discussed.



RW3:

Based on your additional information, then I think that saying that is does not 
allow the gathering of performance data simultaneously is somewhat confusing.  
E.g., you could mak

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Wubo (lana)]

Hi Tom,



Thanks for your comments. Please see inline.



Thanks,

Bo



-Original Message-

From: tom petch [mailto:ie...@btconnect.com]

Sent: Thursday, September 15, 2022 6:37 PM

To: Rob Wilton (rwilton) ; Wubo (lana) 
; draft-ietf-opsawg-yang-vpn-service-pm@ietf.org

Cc: opsawg@ietf.org

Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



From: OPSAWG  on behalf of Rob Wilton (rwilton) 


Sent: 15 September 2022 09:09



Hi Bo,



Looks good.



Let me know when you have an updated version of the draft posted and I will 
kick off the IETF LC.



Thanks for the updates and for taking my comments onboard.





I have been following this thread with a sense of deja vu having made similar 
comments, much on s.4.2 , back in May.  Except, I do not think I ever hit 
'send'.  I was trying to make clear comments that were not confused but found 
the I-D so confusing that I kept on changing my comments to try and make them 
clear and never finished.



My comments were that the document contradicted the Abstract, that the I-D was 
mostly about VPN services and not about network level.  I concluded that this 
I-D was really two separate pieces of work, headed for two separate RFC, banged 
together because they had some groupings in common, and I think that much of 
the discussion in this thread has revolved around that issue.  (It is a bit 
like YANG modules with masses of groupings which save the author repeating a 
few lines of YANG while making it harder for anyone else to follow, except more 
so).



Bo: Thanks for the comments. The authors has designed this model with the 
following considerations:

1) This model is a complementary performance monitoring model for both 
RFC8345, and LxNM (e.g. RFC8299 L3NM ), not just for custom VPN service.

2) Following the design method of RFC8345, this model could be used for 
both underlay networks and overlay VPN networks.

3) The PM metrics and counters of the underlay network are the same as 
those of the VPN network, such as the delay, jitter, and loss of links or 
overlay links (tunnels) and the counters of physical interfaces or VPN access 
interfaces (e.g. VLAN interfaces).



We updated the draft to try to clarify the above, please see

Diff:   
https://www.ietf.org/rfcdiff?url2=draft-ietf-opsawg-yang-vpn-service-pm-10



Thanks,

Bo



So, I shall try to forget what I have learnt from this thread and read the 
revised I-D to see if I find it any clearer but will probably end up with the 
same conclusion, this is two separate RFC.



Tom Petch.



Regards,

Rob





From: Wubo (lana) 

Sent: 15 September 2022 03:17

To: Rob Wilton (rwilton) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org

Cc: opsawg@ietf.org

Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



Hi Rob,



Thank you for the review and helpful comments.



I copied your last comment here, since this is the last point to be discussed.



RW3:

Based on your additional information, then I think that saying that is does not 
allow the gathering of performance data simultaneously is somewhat confusing.  
E.g., you could make a get request that spanned over multiple network list 
entries, or similar for a subscription.



I think that probably nothing extra needs to be said at all.  But if you do 
want to add text here then I suggest that it clarifies that networks and VPNs 
would be separate entries in the network list, and the underlying network would 
not have the “service” container set, whereas the VPN network entries would.



Bo4: Thanks for the suggestion. How about the changes:



==



4.2.  Network Level







The model can be used for performance monitoring both for the network and the 
VPN services. However, the module does not allow to gather the performance 
monitoring data simultaneously for both cases. Concretely: The two would be 
separate entries in the network list. The differences are as follows:



* When the “service-type” presence container is absent, then it indicates



performance monitoring of the network itself.







* When the “service-type” presence container is present, then it indicates



performance monitoring of the VPN service specified by the “service-type”



leaf, e.g. , L3VPN or Virtual Private LAN Service (VPLS).  The values are taken



from [RFC9181].  When a network topology instance contains the L3VPN or



other L2VPN network type, it represents a VPN instance that can perform



performance monitoring.



==



Thanks,

Bo

发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]

发送时间: 2022年9月14日 22:53

收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>

抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>

主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



Hi Bo, authors,



Okay, thanks for the clarifications.  Please see inline …





From: Wubo (l

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Rob Wilton (rwilton)]

My interpretation of the draft was basically this:

(1) The YANG topology model (rfc8345) can model both an underlay network and 
overlaying services.
(2) The base YANG topology model is missing some generic attributes to identify 
that a topology represents a service (e.g., service-type, vpn-id, 
vpn-service-topology).  I don't think that these attributes necessarily have 
anything to with PM, but they were added here because they were needed.  E.g., 
arguably they could have been put into a separate YANG module, but it would 
perhaps be too small to be worthwhile).
(3) The performance monitoring data can largely be gathered either at the 
network layer or at the service layer and this is really distinguished by which 
entry in the topology list the PM data nodes are being returned for.

Authors, is my understanding correct and accurate?

For (2), that does raise a further question:  In section 4.3, the "role" leaf 
has been placed under pm-attributes.  But again, I wonder whether this "role" 
is really just a generic description of the service endpoint.  Hence, would it 
be better to name this "vpn-service-role" and augment it directly under 
/nw:networks/nw:network/nw:node?  Possibly, it could be made conditional on 
/nw:networks/nw:network/nw:network-types/service/service-type

Rob 



> -Original Message-
> From: Adrian Farrel 
> Sent: 16 September 2022 10:34
> To: 'tom petch' ; Rob Wilton (rwilton)
> ; 'Wubo (lana)' ; draft-ietf-
> opsawg-yang-vpn-service-pm....@ietf.org
> Cc: opsawg@ietf.org
> Subject: RE: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-
> pm-09
> 
> Hi Tom, all,
> 
> I think my review as Shepherd ran into the same concern. And it is one of my
> long-standing gripes that "we" (the IETF) repeatedly confuse VPN as a
> service with the means and mechanisms to realise the VPN within the
> network. Of course, as network engineers, it is understandable why we
> make that mistake, but it is also harmful to the way we talk about the
> customers' view of VPNs.
> 
> Now, in discussing this document with the authors, I wanted to distinguish
> between the performance measurement that the customer can perform
> (which is strictly edge-to-edge because the customer cannot see what is
> happening within the network), and the measurements that the provider
> can perform that can be far more analytic about the resources and
> routes/paths within the network. My feeling was that the authors
> completely got this distinction, but that they wanted to look at the
> performance monitoring from the provider's perspective with two
> viewpoints: what can they measure about how their network is performing,
> and what can they measure that will match what the customer might
> measure. Of course, the provider wants to know the latter before the
> customer notices and complains, but the provider also wants to be able to
> link the edge-to-edge measurements back to the more detailed
> measurements from within the network to determine causes.
> 
> It is possible that I have expressed that differently from the way the
> document describes it, and it also possible that I have misrepresented the
> authors and the working group. But that was my take-away.
> 
> Cheers,
> Adrian
> 
> -Original Message-
> From: OPSAWG  On Behalf Of tom petch
> Sent: 15 September 2022 11:37
> To: Rob Wilton (rwilton) ; Wubo (lana)
> ; draft-ietf-opsawg-yang-vpn-service-
> pm@ietf.org
> Cc: opsawg@ietf.org
> Subject: Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-
> pm-09
> 
> From: OPSAWG  on behalf of Rob Wilton
> (rwilton) 
> Sent: 15 September 2022 09:09
> 
> Hi Bo,
> 
> Looks good.
> 
> Let me know when you have an updated version of the draft posted and I
> will kick off the IETF LC.
> 
> Thanks for the updates and for taking my comments onboard.
> 
> 
> I have been following this thread with a sense of deja vu having made similar
> comments, much on s.4.2 , back in May.  Except, I do not think I ever hit
> 'send'.  I was trying to make clear comments that were not confused but
> found the I-D so confusing that I kept on changing my comments to try and
> make them clear and never finished.
> 
> My comments were that the document contradicted the Abstract, that the I-
> D was mostly about VPN services and not about network level.  I concluded
> that this I-D was really two separate pieces of work, headed for two separate
> RFC, banged together because they had some groupings in common, and I
> think that much of the discussion in this thread has revolved around that
> issue.  (It is a bit like YANG modules with masses of groupings which save the
> author repeating a few lines of YANG while making it harder for anyone e

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Adrian Farrel]

Hi Tom, all,

I think my review as Shepherd ran into the same concern. And it is one of my 
long-standing gripes that "we" (the IETF) repeatedly confuse VPN as a service 
with the means and mechanisms to realise the VPN within the network. Of course, 
as network engineers, it is understandable why we make that mistake, but it is 
also harmful to the way we talk about the customers' view of VPNs.

Now, in discussing this document with the authors, I wanted to distinguish 
between the performance measurement that the customer can perform (which is 
strictly edge-to-edge because the customer cannot see what is happening within 
the network), and the measurements that the provider can perform that can be 
far more analytic about the resources and routes/paths within the network. My 
feeling was that the authors completely got this distinction, but that they 
wanted to look at the performance monitoring from the provider's perspective 
with two viewpoints: what can they measure about how their network is 
performing, and what can they measure that will match what the customer might 
measure. Of course, the provider wants to know the latter before the customer 
notices and complains, but the provider also wants to be able to link the 
edge-to-edge measurements back to the more detailed measurements from within 
the network to determine causes.

It is possible that I have expressed that differently from the way the document 
describes it, and it also possible that I have misrepresented the authors and 
the working group. But that was my take-away.

Cheers,
Adrian

-Original Message-
From: OPSAWG  On Behalf Of tom petch
Sent: 15 September 2022 11:37
To: Rob Wilton (rwilton) ; Wubo (lana) 
; draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
Cc: opsawg@ietf.org
Subject: Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

From: OPSAWG  on behalf of Rob Wilton (rwilton) 

Sent: 15 September 2022 09:09

Hi Bo,

Looks good.

Let me know when you have an updated version of the draft posted and I will 
kick off the IETF LC.

Thanks for the updates and for taking my comments onboard.


I have been following this thread with a sense of deja vu having made similar 
comments, much on s.4.2 , back in May.  Except, I do not think I ever hit 
'send'.  I was trying to make clear comments that were not confused but found 
the I-D so confusing that I kept on changing my comments to try and make them 
clear and never finished.

My comments were that the document contradicted the Abstract, that the I-D was 
mostly about VPN services and not about network level.  I concluded that this 
I-D was really two separate pieces of work, headed for two separate RFC, banged 
together because they had some groupings in common, and I think that much of 
the discussion in this thread has revolved around that issue.  (It is a bit 
like YANG modules with masses of groupings which save the author repeating a 
few lines of YANG while making it harder for anyone else to follow, except more 
so).

So, I shall try to forget what I have learnt from this thread and read the 
revised I-D to see if I find it any clearer but will probably end up with the 
same conclusion, this is two separate RFC.

Tom Petch.

Regards,
Rob


From: Wubo (lana) 
Sent: 15 September 2022 03:17
To: Rob Wilton (rwilton) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
Cc: opsawg@ietf.org
Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thank you for the review and helpful comments.

I copied your last comment here, since this is the last point to be discussed.

RW3:
Based on your additional information, then I think that saying that is does not 
allow the gathering of performance data simultaneously is somewhat confusing.  
E.g., you could make a get request that spanned over multiple network list 
entries, or similar for a subscription.

I think that probably nothing extra needs to be said at all.  But if you do 
want to add text here then I suggest that it clarifies that networks and VPNs 
would be separate entries in the network list, and the underlying network would 
not have the “service” container set, whereas the VPN network entries would.

Bo4: Thanks for the suggestion. How about the changes:

==

4.2.  Network Level



The model can be used for performance monitoring both for the network and the 
VPN services. However, the module does not allow to gather the performance 
monitoring data simultaneously for both cases. Concretely: The two would be 
separate entries in the network list. The differences are as follows:

* When the “service-type” presence container is absent, then it indicates

performance monitoring of the network itself.



* When the “service-type” presence container is present, then it indicates

performance monitoring of the VPN service specified by the “service-type”

leaf, e.g. , L3VPN or Virtual Private LAN Service (VPLS).  The values are taken

from [RFC9181].  When a network topolog

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[tom petch]

From: OPSAWG  on behalf of Rob Wilton (rwilton) 

Sent: 15 September 2022 09:09

Hi Bo,

Looks good.

Let me know when you have an updated version of the draft posted and I will 
kick off the IETF LC.

Thanks for the updates and for taking my comments onboard.


I have been following this thread with a sense of deja vu having made similar 
comments, much on s.4.2 , back in May.  Except, I do not think I ever hit 
'send'.  I was trying to make clear comments that were not confused but found 
the I-D so confusing that I kept on changing my comments to try and make them 
clear and never finished.

My comments were that the document contradicted the Abstract, that the I-D was 
mostly about VPN services and not about network level.  I concluded that this 
I-D was really two separate pieces of work, headed for two separate RFC, banged 
together because they had some groupings in common, and I think that much of 
the discussion in this thread has revolved around that issue.  (It is a bit 
like YANG modules with masses of groupings which save the author repeating a 
few lines of YANG while making it harder for anyone else to follow, except more 
so).

So, I shall try to forget what I have learnt from this thread and read the 
revised I-D to see if I find it any clearer but will probably end up with the 
same conclusion, this is two separate RFC.

Tom Petch.

Regards,
Rob


From: Wubo (lana) 
Sent: 15 September 2022 03:17
To: Rob Wilton (rwilton) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
Cc: opsawg@ietf.org
Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thank you for the review and helpful comments.

I copied your last comment here, since this is the last point to be discussed.

RW3:
Based on your additional information, then I think that saying that is does not 
allow the gathering of performance data simultaneously is somewhat confusing.  
E.g., you could make a get request that spanned over multiple network list 
entries, or similar for a subscription.

I think that probably nothing extra needs to be said at all.  But if you do 
want to add text here then I suggest that it clarifies that networks and VPNs 
would be separate entries in the network list, and the underlying network would 
not have the “service” container set, whereas the VPN network entries would.

Bo4: Thanks for the suggestion. How about the changes:

==

4.2.  Network Level



The model can be used for performance monitoring both for the network and the 
VPN services. However, the module does not allow to gather the performance 
monitoring data simultaneously for both cases. Concretely: The two would be 
separate entries in the network list. The differences are as follows:

* When the “service-type” presence container is absent, then it indicates

performance monitoring of the network itself.



* When the “service-type” presence container is present, then it indicates

performance monitoring of the VPN service specified by the “service-type”

leaf, e.g. , L3VPN or Virtual Private LAN Service (VPLS).  The values are taken

from [RFC9181].  When a network topology instance contains the L3VPN or

other L2VPN network type, it represents a VPN instance that can perform

performance monitoring.

==

Thanks,
Bo
发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月14日 22:53
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo, authors,

Okay, thanks for the clarifications.  Please see inline …


From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 14 September 2022 15:31
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thanks again for your review.  Please find our reply inline.

Thanks,
Bo

发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月14日 17:18
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo, authors,

Please see inline. Again, I have removed sections where we have agreement.  I 
think that there is just one area that I’m still slightly confused by relating 
to the network vs service PM, for which I’ve added some further questions 
inline.



From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 14 September 2022 09:25
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mai

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Rob Wilton (rwilton)]

Hi Bo,

Looks good.

Let me know when you have an updated version of the draft posted and I will 
kick off the IETF LC.

Thanks for the updates and for taking my comments onboard.

Regards,
Rob


From: Wubo (lana) 
Sent: 15 September 2022 03:17
To: Rob Wilton (rwilton) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
Cc: opsawg@ietf.org
Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thank you for the review and helpful comments.

I copied your last comment here, since this is the last point to be discussed.

RW3:
Based on your additional information, then I think that saying that is does not 
allow the gathering of performance data simultaneously is somewhat confusing.  
E.g., you could make a get request that spanned over multiple network list 
entries, or similar for a subscription.

I think that probably nothing extra needs to be said at all.  But if you do 
want to add text here then I suggest that it clarifies that networks and VPNs 
would be separate entries in the network list, and the underlying network would 
not have the “service” container set, whereas the VPN network entries would.

Bo4: Thanks for the suggestion. How about the changes:

==

4.2.  Network Level



The model can be used for performance monitoring both for the network and the 
VPN services. However, the module does not allow to gather the performance 
monitoring data simultaneously for both cases. Concretely: The two would be 
separate entries in the network list. The differences are as follows:

* When the “service-type” presence container is absent, then it indicates

performance monitoring of the network itself.



* When the “service-type” presence container is present, then it indicates

performance monitoring of the VPN service specified by the “service-type”

leaf, e.g. , L3VPN or Virtual Private LAN Service (VPLS).  The values are taken

from [RFC9181].  When a network topology instance contains the L3VPN or

other L2VPN network type, it represents a VPN instance that can perform

performance monitoring.

==

Thanks,
Bo
发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月14日 22:53
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo, authors,

Okay, thanks for the clarifications.  Please see inline …


From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 14 September 2022 15:31
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thanks again for your review.  Please find our reply inline.

Thanks,
Bo

发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月14日 17:18
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo, authors,

Please see inline. Again, I have removed sections where we have agreement.  I 
think that there is just one area that I’m still slightly confused by relating 
to the network vs service PM, for which I’ve added some further questions 
inline.



From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 14 September 2022 09:25
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thank again for your deep review. Please find our response inline for the open 
points.

Best regards,
Bo


发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月13日 17:24
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo,

Thanks.  I’ve made some further comments for a few points inline.  I’ve snipped 
those that we already have agreement on.


From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 13 September 2022 07:38
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09


Hi Rob,



Many thanks for your thou

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Wubo (lana)]

Hi Rob,

Thank you for the review and helpful comments.

I copied your last comment here, since this is the last point to be discussed.

RW3:
Based on your additional information, then I think that saying that is does not 
allow the gathering of performance data simultaneously is somewhat confusing.  
E.g., you could make a get request that spanned over multiple network list 
entries, or similar for a subscription.

I think that probably nothing extra needs to be said at all.  But if you do 
want to add text here then I suggest that it clarifies that networks and VPNs 
would be separate entries in the network list, and the underlying network would 
not have the “service” container set, whereas the VPN network entries would.

Bo4: Thanks for the suggestion. How about the changes:

==

4.2.  Network Level



The model can be used for performance monitoring both for the network and the 
VPN services. However, the module does not allow to gather the performance 
monitoring data simultaneously for both cases. Concretely: The two would be 
separate entries in the network list. The differences are as follows:

* When the “service-type” presence container is absent, then it indicates

performance monitoring of the network itself.



* When the “service-type” presence container is present, then it indicates

performance monitoring of the VPN service specified by the “service-type”

leaf, e.g. , L3VPN or Virtual Private LAN Service (VPLS).  The values are taken

from [RFC9181].  When a network topology instance contains the L3VPN or

other L2VPN network type, it represents a VPN instance that can perform

performance monitoring.

==

Thanks,
Bo
发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月14日 22:53
收件人: Wubo (lana) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
抄送: opsawg@ietf.org
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo, authors,

Okay, thanks for the clarifications.  Please see inline …


From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 14 September 2022 15:31
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thanks again for your review.  Please find our reply inline.

Thanks,
Bo

发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月14日 17:18
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo, authors,

Please see inline. Again, I have removed sections where we have agreement.  I 
think that there is just one area that I’m still slightly confused by relating 
to the network vs service PM, for which I’ve added some further questions 
inline.



From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 14 September 2022 09:25
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thank again for your deep review. Please find our response inline for the open 
points.

Best regards,
Bo


发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月13日 17:24
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm....@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo,

Thanks.  I’ve made some further comments for a few points inline.  I’ve snipped 
those that we already have agreement on.


From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 13 September 2022 07:38
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09


Hi Rob,



Many thanks for your thoughtful review. Please see inline.



Thanks,



Bo



-邮件原件-
发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月9日 18:43
收件人: 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



Hi,



Here are my AD review comments for draft-ietf-opsawg-yang-vpn-service-pm-09, 
apologies for the delay.



I think that this document is in good shape and h

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Rob Wilton (rwilton)]

Hi Bo, authors,

Okay, thanks for the clarifications.  Please see inline …


From: Wubo (lana) 
Sent: 14 September 2022 15:31
To: Rob Wilton (rwilton) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
Cc: opsawg@ietf.org
Subject: Re: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thanks again for your review.  Please find our reply inline.

Thanks,
Bo

发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月14日 17:18
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo, authors,

Please see inline. Again, I have removed sections where we have agreement.  I 
think that there is just one area that I’m still slightly confused by relating 
to the network vs service PM, for which I’ve added some further questions 
inline.



From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 14 September 2022 09:25
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thank again for your deep review. Please find our response inline for the open 
points.

Best regards,
Bo


发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月13日 17:24
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo,

Thanks.  I’ve made some further comments for a few points inline.  I’ve snipped 
those that we already have agreement on.


From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 13 September 2022 07:38
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09


Hi Rob,



Many thanks for your thoughtful review. Please see inline.



Thanks,



Bo



-邮件原件-
发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月9日 18:43
收件人: 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



Hi,



Here are my AD review comments for draft-ietf-opsawg-yang-vpn-service-pm-09, 
apologies for the delay.



I think that this document is in good shape and hence most of my comments are 
only minor or nits.







(11) p 8, sec 4.2.  Network Level



   For network performance monitoring, the container of "networks" in

   [RFC8345] is not extended.



I'm confused by what this sentence is meant to convey - did you mean augmented? 
 In particular, it isn't clear to me how you express PM for the physical (or 
underlay networks).  Is what you are trying to express that the "service-type" 
container is present for VPN service performance monitoring and absence 
otherwise?  Probably more words required here, and in the YANG module.



Bo: Thanks for pointing this out. Your understanding is exactly what we're 
trying to convey. How about we change to



As VPN Network PM YANG module includes two types of PM augmentation, the 
underlay networks PM is augmented on [RFC8345] when the "service-type" presence 
container is not defined

, and the VPN PM is augmented on [RFC8345] when the "service-type" presence 
container is defined.



For the underlay network performance monitoring, the container of "networks" in

   [RFC8345] is not augmented.



I think that I would still find that slightly confusing.  Perhaps:



NEW:



4.2.  Network Level



The model can be used for performance monitoring both for the network and the 
VPN services.



When the “service-type” presence container is absent, then it indicates

performance monitoring of the network itself.



When the “service-type” presence container is present, then it indicates

performance monitoring of the VPN service specified by the “service-type”

leaf, e.g. , L3VPN or Virtual Private LAN Service (VPLS).  The values are taken

from [RFC9181].  When a network topology instance contains the L3VPN or

other L2VPN network type, it represents a VPN instance that can perform

performance monitoring.


Bo 2: Thanks for the good suggestion. The text looks good.



One extra question:



Does this model allow you to gather PM data from both the network and L2VPN 
services at the same tim

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Wubo (lana)]

Hi Rob,

Thanks again for your review.  Please find our reply inline.

Thanks,
Bo

发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月14日 17:18
收件人: Wubo (lana) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
抄送: opsawg@ietf.org
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo, authors,

Please see inline. Again, I have removed sections where we have agreement.  I 
think that there is just one area that I’m still slightly confused by relating 
to the network vs service PM, for which I’ve added some further questions 
inline.



From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 14 September 2022 09:25
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thank again for your deep review. Please find our response inline for the open 
points.

Best regards,
Bo


发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月13日 17:24
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo,

Thanks.  I’ve made some further comments for a few points inline.  I’ve snipped 
those that we already have agreement on.


From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 13 September 2022 07:38
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09


Hi Rob,



Many thanks for your thoughtful review. Please see inline.



Thanks,



Bo



-邮件原件-
发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月9日 18:43
收件人: 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



Hi,



Here are my AD review comments for draft-ietf-opsawg-yang-vpn-service-pm-09, 
apologies for the delay.



I think that this document is in good shape and hence most of my comments are 
only minor or nits.







(11) p 8, sec 4.2.  Network Level



   For network performance monitoring, the container of "networks" in

   [RFC8345] is not extended.



I'm confused by what this sentence is meant to convey - did you mean augmented? 
 In particular, it isn't clear to me how you express PM for the physical (or 
underlay networks).  Is what you are trying to express that the "service-type" 
container is present for VPN service performance monitoring and absence 
otherwise?  Probably more words required here, and in the YANG module.



Bo: Thanks for pointing this out. Your understanding is exactly what we're 
trying to convey. How about we change to



As VPN Network PM YANG module includes two types of PM augmentation, the 
underlay networks PM is augmented on [RFC8345] when the "service-type" presence 
container is not defined

, and the VPN PM is augmented on [RFC8345] when the "service-type" presence 
container is defined.



For the underlay network performance monitoring, the container of "networks" in

   [RFC8345] is not augmented.



I think that I would still find that slightly confusing.  Perhaps:



NEW:



4.2.  Network Level



The model can be used for performance monitoring both for the network and the 
VPN services.



When the “service-type” presence container is absent, then it indicates

performance monitoring of the network itself.



When the “service-type” presence container is present, then it indicates

performance monitoring of the VPN service specified by the “service-type”

leaf, e.g. , L3VPN or Virtual Private LAN Service (VPLS).  The values are taken

from [RFC9181].  When a network topology instance contains the L3VPN or

other L2VPN network type, it represents a VPN instance that can perform

performance monitoring.


Bo 2: Thanks for the good suggestion. The text looks good.



One extra question:



Does this model allow you to gather PM data from both the network and L2VPN 
services at the same time?  If so, is there, or should there be, any text is 
the document that describes how to do this?


Bo2: In the current model design, the underlay network and L2VPN are separate 
network instances and the PM data cannot be gathered at the same time.

RW2:
Okay.  I would like to dig into this one a bit more, to understand whether this 
is a real limitation or not, and to ensure that I understand the model 
correctly:

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Rob Wilton (rwilton)]

Hi Bo, authors,

Please see inline. Again, I have removed sections where we have agreement.  I 
think that there is just one area that I’m still slightly confused by relating 
to the network vs service PM, for which I’ve added some further questions 
inline.



From: Wubo (lana) 
Sent: 14 September 2022 09:25
To: Rob Wilton (rwilton) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
Cc: opsawg@ietf.org
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Rob,

Thank again for your deep review. Please find our response inline for the open 
points.

Best regards,
Bo


发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月13日 17:24
收件人: Wubo (lana) mailto:lana.w...@huawei.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: RE: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

Hi Bo,

Thanks.  I’ve made some further comments for a few points inline.  I’ve snipped 
those that we already have agreement on.


From: Wubo (lana) mailto:lana.w...@huawei.com>>
Sent: 13 September 2022 07:38
To: Rob Wilton (rwilton) mailto:rwil...@cisco.com>>; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
Cc: opsawg@ietf.org<mailto:opsawg@ietf.org>
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09


Hi Rob,



Many thanks for your thoughtful review. Please see inline.



Thanks,



Bo



-邮件原件-
发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月9日 18:43
收件人: 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org<mailto:draft-ietf-opsawg-yang-vpn-service-pm@ietf.org>
抄送: opsawg@ietf.org<mailto:opsawg@ietf.org>
主题: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



Hi,



Here are my AD review comments for draft-ietf-opsawg-yang-vpn-service-pm-09, 
apologies for the delay.



I think that this document is in good shape and hence most of my comments are 
only minor or nits.







(11) p 8, sec 4.2.  Network Level



   For network performance monitoring, the container of "networks" in

   [RFC8345] is not extended.



I'm confused by what this sentence is meant to convey - did you mean augmented? 
 In particular, it isn't clear to me how you express PM for the physical (or 
underlay networks).  Is what you are trying to express that the "service-type" 
container is present for VPN service performance monitoring and absence 
otherwise?  Probably more words required here, and in the YANG module.



Bo: Thanks for pointing this out. Your understanding is exactly what we're 
trying to convey. How about we change to



As VPN Network PM YANG module includes two types of PM augmentation, the 
underlay networks PM is augmented on [RFC8345] when the "service-type" presence 
container is not defined

, and the VPN PM is augmented on [RFC8345] when the "service-type" presence 
container is defined.



For the underlay network performance monitoring, the container of "networks" in

   [RFC8345] is not augmented.



I think that I would still find that slightly confusing.  Perhaps:



NEW:



4.2.  Network Level



The model can be used for performance monitoring both for the network and the 
VPN services.



When the “service-type” presence container is absent, then it indicates

performance monitoring of the network itself.



When the “service-type” presence container is present, then it indicates

performance monitoring of the VPN service specified by the “service-type”

leaf, e.g. , L3VPN or Virtual Private LAN Service (VPLS).  The values are taken

from [RFC9181].  When a network topology instance contains the L3VPN or

other L2VPN network type, it represents a VPN instance that can perform

performance monitoring.


Bo 2: Thanks for the good suggestion. The text looks good.



One extra question:



Does this model allow you to gather PM data from both the network and L2VPN 
services at the same time?  If so, is there, or should there be, any text is 
the document that describes how to do this?


Bo2: In the current model design, the underlay network and L2VPN are separate 
network instances and the PM data cannot be gathered at the same time.

RW2:
Okay.  I would like to dig into this one a bit more, to understand whether this 
is a real limitation or not, and to ensure that I understand the model 
correctly:

I’m not really concerned about whether the data can be gathered at the same 
time (i.e., in the same request), but I would have thought that it is likely 
that some operators may want to do PM at both the network and overlay at the 
same time.

If you take the diagram in 4.1, that shows an underlay network with two VPN1 
and VPN2 service overlays, then am I right to assume that they will be modelled 
as 3 separate list entries in the /nw:networks/nw:network/ list, one 

Re: [OPSAWG] AD review of draft-ietf-opsawg-yang-vpn-service-pm-09

[Rob Wilton (rwilton)]

Hi Bo,

Thanks.  I’ve made some further comments for a few points inline.  I’ve snipped 
those that we already have agreement on.


From: Wubo (lana) 
Sent: 13 September 2022 07:38
To: Rob Wilton (rwilton) ; 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
Cc: opsawg@ietf.org
Subject: 答复: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09


Hi Rob,



Many thanks for your thoughtful review. Please see inline.



Thanks,



Bo



-邮件原件-
发件人: Rob Wilton (rwilton) [mailto:rwil...@cisco.com]
发送时间: 2022年9月9日 18:43
收件人: 
draft-ietf-opsawg-yang-vpn-service-pm@ietf.org
抄送: opsawg@ietf.org
主题: AD review of draft-ietf-opsawg-yang-vpn-service-pm-09



Hi,



Here are my AD review comments for draft-ietf-opsawg-yang-vpn-service-pm-09, 
apologies for the delay.



I think that this document is in good shape and hence most of my comments are 
only minor or nits.





Minor level comments:



(1) p 0, sec



   The data model for network topologies defined in RFC 8345 introduces

   vertical layering relationships between networks that can be

   augmented to cover network and service topologies.  This document

   defines a YANG module for performance monitoring (PM) of both

  networks and VPN services that can be used to monitor and manage

   network performance on the topology at higher layer or the service

   topology between VPN sites.



"the topology at higher layer" doesn't scan particularly well to me, please can 
you tweak it.



Bo: Thanks for pointing this out. Is this better that we simply change to “the 
underlay topology”?



Yes, perhaps something like this:



NEW:

   The data model for network topologies defined in RFC 8345 introduces

   vertical layering relationships between networks that can be

   augmented to cover network and service topologies.  This document

   defines a YANG module for performance monitoring (PM) of both

   underlay networks and overlay VPN services that can be used to monitor

  and manage network performance on the topology of both layers.







(3) p 4, sec 3.  Network and VPN Service Performance Monitoring Model Usage



   As shown in Figure 1, in the context of the layering model

   architecture described in [RFC8309], the network and VPN service

   performance monitoring (PM) model can be used to expose a set of

   performance information to the above layer.  Such information can be

   used by an orchestrator to subscribe to performance data.



Perhaps rephase?  I.e., is it the performance data that is being used to create 
a subscription based on the performance data, or is it just that the model 
makes the performance data readily available, which can then be subscribed do?



Bo: Thanks for the suggestion. How about:

The model makes the performance data readily available, which can then be 
subscribed by the client application, such as an orchestrator.



I think that you can probably get away with deleting the last 2 sentences of 
that paragraph and rewording it slightly.  The document already talks more 
about the specifics in sections 3.1 and 3.2 anyway.  Hence, I propose:



OLD:



   As shown in Figure 1, in the context of the layering model

   architecture described in [RFC8309], the network and VPN service

   performance monitoring (PM) model can be used to expose a set of

   performance information to the above layer.  Such information can be

   used by an orchestrator to subscribe to performance data.  The

   network controller will then notify the orchestrator about

   corresponding parameter changes.



NEW:



As shown in Figure 1, in the context of the layering model

architecture described in [RFC8309], the network and VPN service

performance monitoring (PM) model can be used to expose operational

performance information to the layer above, e.g., to an orchestrator

or other client application, via standard network management APIs.









(5) p 5, sec 3.1.  Collecting Data via Pub/Sub Mechanism



   Some applications such as service-assurance applications, which must

   maintain a continuous view of operational data and state, can use the

   subscription model specified in [RFC8641] to subscribe to the

   specific network performance data or VPN service performance data

   they are interested in, at the data source.  For example, network or

   VPN topology updates may be obtained through on-change notifications

   [RFC8641].  For dynamic PM data, various notifications can be

   specified to obtain more complete data.



Can you elaborate a bit on what is meant by dynamic PM data please.



Bo: Thanks for pointing this out. How about we change:

For dynamic PM data, e.g. VRF routes or MAC entries, link metrics, and 
interface metrics, various notifications can be specified to obtain more 
complete data.



Looks good.  Thanks.





(6) p 5, sec 3.1.  Collecting Data via Pub/Sub Mechanism



  A periodic notification