As you can see from the lengthy discussion below, VPP is filling yet another need and so is P4.
Hemant From: hemant via p4-design <p4-des...@lists.p4.org> Sent: Tuesday, August 09, 2022 9:46 AM To: 'Gergely Pongracz' <gergely.pongr...@ericsson.com>; mbu...@vmware.com; jnfos...@cs.cornell.edu Cc: p4-des...@lists.p4.org Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Got it, thanks. The Cavium/Marvell Octeon has been used for your Tofino tasks but the Octeon is not P4 programmable. The Octeon is flexible with 16, 32, or 42 cores. If your need is 1 Tbps or little less, VPP can run IPv4 crypto on x86 at 1 Tbps on a single machine. While you wait for a smaller Tofino or the Octeon to be P4 programmable, consider VPP because point-to-point flows will run faster in VPP than dpdk. Spinning a new asic or writing a new p4c backend for Octeon are long poles. If I want to ship today, VPP works and a P4toVPP compiler and tools are available from my company. I also said our compiler supports P4toDPDK by running one packet in VPP vector. Additionally, since VPP uses dpdk for packet i/o , one gets HQF qos from dpdk – no switching asic supports HQF and HQF is the only means I know of to provide qos for individual UE flows. Hemant From: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> > Sent: Tuesday, August 09, 2022 4:54 AM To: hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; mbu...@vmware.com <mailto:mbu...@vmware.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Hi, What I meant is that Tofino supports much higher bandwidth compared to our needs. If one would design a “small Tofino”, e.g., around terabit/sec (maybe even under) that consume (much) less power and costs less that would be ideal for this task. Gergely From: hem...@mnkcg.com <mailto:hem...@mnkcg.com> <hem...@mnkcg.com <mailto:hem...@mnkcg.com> > Sent: Friday, August 5, 2022 9:15 PM To: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> >; mbu...@vmware.com <mailto:mbu...@vmware.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Thanks for the details. Why was Tofino not very ideal? Hemant From: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> > Sent: Friday, August 05, 2022 4:46 AM To: hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; mbu...@vmware.com <mailto:mbu...@vmware.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Hi, The real base station asic is pretty much a DSP as most of the task is to en/decode signals. But I don’t know the latest details about the bst hardware. In our prototype we decoupled the signal processing part from the rest, so in our case we used a Tofino (not very ideal) for the packet processing tasks + a CPU (or a k8s service) for buffering / retransmission. Tofino code was in P4, while the CPU code was in C (using the dpdk libraries). Gergely From: hem...@mnkcg.com <mailto:hem...@mnkcg.com> <hem...@mnkcg.com <mailto:hem...@mnkcg.com> > Sent: Thursday, August 4, 2022 3:51 PM To: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> >; mbu...@vmware.com <mailto:mbu...@vmware.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Thanks for the clarifications. Seems like a smartNIC would work for retransmit buffer. In your slides you say a SmartSwitch runs RLC at 5000 Mpps – what asic did the switch use and how did you program the asic? Used C? Hemant From: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> > Sent: Thursday, August 04, 2022 5:19 AM To: hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; mbu...@vmware.com <mailto:mbu...@vmware.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Hi, Thanks for resurrecting this old thread, nice surprise. I think this 65k limit can be made less strict in the coming releases (e.g., 6G timeframe) if we have good arguments for that. One good argument could be that it can’t be supported in asics. But there is one misunderstanding. This is a retransmit buffer only, so no segmentation / reassembly is needed. Regarding the number of packets: since this is a retransmit loop it is basically characterized by the over the air RTT and the bandwidth. In 5G the typical values are 5-10 msec and a few Gbps per cell. So we’ll need 1-2 MB space for the retransmit buffer. And a typical base station handles multiple cells, so the buffer will be in the few tens of MB range. I do agree that this is not a suitable task for a Tofino-like device. In RAN we don’t have terabits per sec bandwidth, but we might want to do more processing per packet. A CPU on the other hand seems a bit too heavy for this. BR, Gergely From: hem...@mnkcg.com <mailto:hem...@mnkcg.com> <hem...@mnkcg.com <mailto:hem...@mnkcg.com> > Sent: Tuesday, July 26, 2022 10:27 PM To: mbu...@vmware.com <mailto:mbu...@vmware.com> ; Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> >; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Today, I thought some more about this one. 3GPP standards mandate up to 65k packet buffering which no switching asic running at 12 Tbps can support. Further, buffering a few packets for reassembly of fragmented packets is possible in asic but, reassembly is an end host function – why do reassembly in a switching asic? Even a cpu system such as VPP has a tight processing budget. VPP runs two instructions in one clock cycle across the board. One could use a high end FPGA for this work. But P4 to FPGA synthesis has some limitations for what P4 subset is supported. The cpu system seems to be the only choice. VPP supports a periodic process for any plugin and the process processes events defined in an enum. So timer can be avoided. Further, since VPP uses DPDK for packet i/o, if one uses one packet in VPP vector, one is running DPDK with benefits of VPP CLI, tracing, and control plane. Regarding P4toVPP, my company has a compiler, https://mnkcg.com/products/p4-to-vpp-compiler/ <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-8a27d19ae6da1d38&q=1&e=f8d8bc7f-90d5-4fad-952f-0343b29ac19c&u=https%3A%2F%2Fmnkcg.com%2Fproducts%2Fp4-to-vpp-compiler%2F> . Hemant From: Mihai Budiu <mbu...@vmware.com <mailto:mbu...@vmware.com> > Sent: Friday, June 18, 2021 4:36 PM F To: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> >; hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases A header is defined by a P4 parser. If you are willing to parse an entire packet then you won’t have a payload. But in general, you can’t write parsers for arbitrary-size packets, even using varbits – in the extract call you have to *know* the header size that you want to parse. Moreover, hardware devices like Tofino have a limit on both the number of parser transitions that they can execute for a packet and the number of bytes parsed per state. These are very natural, if you expect N packets/second/parser and you need P parser transitions per packet, then you need to perform P * N * clock period transitions/second to parse the packets (a device like Tofino has fewer parsers than input ports). This puts a bound on P if you want to sustain this bandwidth without dropping packets (or buffering them). You can statically bound P for the worst case, and reject programs that have parsers that are too complex, or assume something about the expected duration P for the mix of packets you are getting (P is not a constant, it depends on the packet) and hope that you have enough parser bandwidth to cope with all packets in practice. For a NIC N is much smaller than for a router, so it may be feasible to do complex parsing, but for a high speed switch you really don’t have enough time to do deep parsing at line rate. Mihai From: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> > Sent: Friday, June 18, 2021 2:06 AM To: Mihai Budiu <mbu...@vmware.com <mailto:mbu...@vmware.com> >; hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases I guess the “only headers can be stored” could work as long as that header can be 1,5k byte long. 😉 Unfortunately I guess this is not the case, so my proposed workaround doesn’t really work generally. Then as Hemant said: we’d need something (extern, language construct) to be able to work on the payload. Or we’d need to increase the size of the header structure to 1,5k (this way jumbo frames will still cause problems, but the majority of the use cases would work). Actually I think payload buffer would be simpler, as that would require less functionality (basically store, send, delete), but if we could extend the header that would open up interesting possibilities, e.g. http/sip parsing, proxies, DPI. BR, Gergely From: Mihai Budiu <mbu...@vmware.com <mailto:mbu...@vmware.com> > Sent: Friday, June 18, 2021 2:17 AM To: hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> >; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Packet_in is an extern. There are no operations on extern except instantiation and method calls. In particular, there is no assignment between externs (except compile-time binding as parameters). If you want to do something like this you will probably have to invent a new extern to represent a dynamic array. Mihai From: hem...@mnkcg.com <mailto:hem...@mnkcg.com> <hem...@mnkcg.com <mailto:hem...@mnkcg.com> > Sent: Thursday, June 17, 2021 8:24 AM To: hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; Mihai Budiu <mbu...@vmware.com <mailto:mbu...@vmware.com> >; gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases To store packets, I would need an array of Packet_in in P4 which is a new construct for P4. Hemant From: Hemant Singh via P4-design <p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> > Sent: Wednesday, June 16, 2021 7:37 PM To: mbu...@vmware.com <mailto:mbu...@vmware.com> ; gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: [P4-design] Re: new language features for "5G" and "edge" use cases If a packet is not stored in a register, define a new BaaS control which has one arg as packet_in. I think P4 would need to define a new data struct to store whole packets. Hemant From: Mihai Budiu <mbu...@vmware.com <mailto:mbu...@vmware.com> > Sent: Wednesday, June 16, 2021 3:13 PM To: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> >; hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Shouldn’t we have this conversation on the design mailing list? More people may want to weigh in. We have been making moves towards factoring psa.p4 and pna.p4 into a set of common libraries, e.g., common externs (like registers). Other libraries will be useful, like standard protocol header definitions. So I think that in the long term there will be a set of useful libraries, e.g., timers, buffers, etc., and a target architecture may include several of them, signaling in this way what is available. You cannot store the payload of a packet in a register, only the headers. Mihai From: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> > Sent: Wednesday, June 16, 2021 2:31 AM To: Mihai Budiu <mbu...@vmware.com <mailto:mbu...@vmware.com> >; hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases Hi, I think it would make sense to think whether such functionality could be implemented on non-CPU systems or not. If not (always) then I fully agree to have them as externs. But it would be of course in my view better to have them as part of PSA or PNA or some future, widely adopted architecture. Of course these calls will use “extern” functions in the underlying system, but having them as part of a few generic architectures would be good because that would mean better portability of the P4 codes. Having buffers in a Tofino-like device doesn’t seem to be impossible, as Hemant pointed out below one could use register arrays for that purpose. Of course the size is limited, but the functionality seems to be there. I guess the issue is more how we implement callbacks on a Tofino-like device. One workaround could be to have a register array with the packets and timeout events just how Hemant described below and on each (or each Nth) packet arrival we’d check the arrival time and compare it with the timer in the first entry of the reg array. If we exceeded the timer we’d clone the packet and would do the timer based event on the copy (basically dropping the original packet and getting the buffered one to work on). This could fly for one specific timeout (e.g. a fix 100 msec resend timer), in which case the register array contains events as an ordered list (well, better say a ring buffer, but still the timestamps are constantly ascending). Of course this is nasty and should be hidden under some nice API, I just wanted to check whether it could work or not. Gergely From: Mihai Budiu <mbu...@vmware.com <mailto:mbu...@vmware.com> > Sent: Monday, June 14, 2021 7:55 PM To: hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> >; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Subject: RE: [P4-design] Re: new language features for "5G" and "edge" use cases I don’t know of any programming language where timers are a built-in construct, they are always library functions, i.e., externs for us. So I don’t see a choice. Perhaps the question is “which architecture/library file should the timers be a part of?” Mihai From: Hemant Singh via P4-design <p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> > Sent: Friday, June 11, 2021 4:11 PM To: hem...@mnkcg.com <mailto:hem...@mnkcg.com> ; gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: [P4-design] Re: new language features for "5G" and "edge" use cases I don’t see a way out unless start and stop timer are added as new externs. Gergely doesn’t like externs because the code is not portable. But the new externs can be added for all software p4c backends. If externs are not used, what other choice do we have? Maybe we can discuss this question in the June 14th LDWG meeting. Thanks, Hemant From: Hemant Singh via P4-design <p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> > Sent: Thursday, June 10, 2021 11:52 AM To: gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> ; jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> Cc: p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> Subject: [P4-design] Re: new language features for "5G" and "edge" use cases Here is strawman P4 events code to implement BaaS. shared_register is an extern used by the Ibanez paper enum Events { set_timer, del_timer, exp_timer } struct metadata_t { Events ev; … } The parser sets events for set_timer and del_timer. When set_timer is used, a callback is registered which sets exp_timer event. shared_register <bit <32 > >( NUM_REGS ) bufSize_reg ; bit<64> expire_time = 2; if (meta.ev == Events.set_timer) { meta.handle = timer_start(expire_time); bufSize_reg . write (handle , hdr); } else if (meta.ev == Events.del_timer) { timer_stop(meta.handle); bufSize_reg . write (handle , 0 ); } else if (meta.ev == Events.exp_timer) { resend(hdr); meta.handle = timer_start(expire_time); bufSize_reg . write (handle , hdr); } We will need time_start() and timer_stop() service from a Timer block similar to Traffic Manager in one architecture. The code can be extended to support multiple timers. Right now, the code uses one timer and thus the single meta.handle is used. Hemant From: Gergely Pongracz via P4-design <p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> > Sent: Friday, June 04, 2021 5:14 AM To: Nate Foster <jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> > Cc: p4-design <p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> > Subject: [P4-design] Re: new language features for "5G" and "edge" use cases Hi Nate, Sorry for the long delay. I uploaded our example code here: https://github.com/P4ELTE/use_cases/tree/master/p4-16/bst The main code is cpf_ran.p4. In theory it supports both TNA, PSA and V1 – we used Tofino’s compiler and t4p4s (basically p4c with DPDK backend) for compiling and running. Buffering would be executed in the RANDownlink() control block, now we add a special header and send out the packet towards the service IP of the BaaS (buffer-as-a-service) which runs as a Kubernetes service for now. We could clone the packet just as well and send it directly to the downlink path while sending the copy to the buffer, but now it is sent to the buffer and on successful buffering the BaaS service returns the packet – this way we know that the original packet is buffered and timeout counter is started. Regarding to your questions: 1. You are right, maybe it could be solved by an extern similarly as we solve it with a non-P4 component. On the other hand I don’t particularly like having too much architectures around as that kills one of the main advantages of P4 (to my knowledge) which is portability. So I’d rather go for a language change with this – for me the only reason not doing that could be if the task would be impossible to support by some hardware targets. You know the language much better, but I’d say buffering a few packets could be similar to having a bit more registers. So buffering itself doesn’t seem a huge issue for me. Running timers and assigning events to them on the other hand might be a bigger change as potentially there would be a large amount of parallel timers – and of course there are good data structures for that, but are they hardware friendly enough? Ibanez’s presentation suggests it can be done fairly simply on FPGA. 2. According to the presentation I think the proposed solution – especially if all proposed primitives on slide 6 would be implemented – is a superset of what we’d need (I’d say for us enqueue, dequeue and timer expiration would be enough). So if Ibanez’s proposal would be part of the language, we wouldn’t need more (at least for now). 3. Yes, if you have a look at the code you’ll see that we already use control blocks for modularizing the code. With Tofino sometimes it’s not straightforward as the compiler tends to use more stages in this case compared to if you use less control blocks (this issue was also mentioned in the uP4 talk). As I understood, Lyra is a higher layer solution for portability over multiple DSLs, so I guess that would be handy if even in the long term portability would be an issue. I think Lyra’s composition part could deal with composing multiple modules / programs on a single switch – I guess you referred to this feature, but I don’t think we’d need a Lyra-like engine in the long run. So my only question that is remaining: is the proposals from Ibanez & co. already considered by some of the working groups e.g. LDWG? If yes, I’ll go thru the details as that is quite likely a good solution for us too. Thanks! Gergely From: Nate Foster <jnfos...@cs.cornell.edu <mailto:jnfos...@cs.cornell.edu> > Sent: Tuesday, June 1, 2021 3:51 PM To: Gergely Pongracz <gergely.pongr...@ericsson.com <mailto:gergely.pongr...@ericsson.com> > Cc: p4-design <p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> > Subject: Re: [P4-design] new language features for "5G" and "edge" use cases Hi Gergely, The best way to get involved is to attend the monthly P4 LDWG meeting. For bringing proposals, we have a process discussed on the README.md <https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-ab8d194f3024e5fc&q=1&e=befde3df-5fb2-40c1-9541-7f96681c5d36&u=http%3A%2F%2Freadme.md%2F> for the p4-spec repository (https://github.com/p4lang/p4-spec). Basically we are happy to entertain a high-level proposal, resulting in a thumbs up or thumbs down. For detailed proposals, we expect to see a number of things fully worked out: specification language changes, prototype implementation, and example programs. Suffice to say, this is a lot of work, so it's great to either be certain you want to pursue it and are able to follow through, or you can convince others to help you out. Responding to these topics: 1. Does this need a language change or could these buffers be modeled with an extern? If so, then no language change is needed, just an architecture that supports buffering. 2. Is the idea the same as Ibanez et al.'s notion of events (https://dl.acm.org/doi/10.1145/3365609.3365848)? If not, what language changes would you propose? 3. Simple forms of modularity can be accomplished by treating controls as composable units. Note that they support constructors with statically-determined parameters. Otherwise, have you looked at Lyra and MicroP4 from SIGCOMM '20? (Minor: we have shifted to inclusive technology, so I recommend using the name "primary" and not the one you used.) -N On Tue, Jun 1, 2021 at 8:56 AM Gergely Pongracz via P4-design <p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> > wrote: Hi, On the P4 WS we presented a use case that is I admit not very unique these days (implementing 5G network functions with P4), but while we were implementing these NFs we came across a few limitations in P4 and started to wonder whether some new features solving these issues could be part of P4 in the future. There are a few slides on these: https://opennetworking.org/wp-content/uploads/2021/05/Gergely-Pongracz-Slides.pdf Basically there are the following 3 cases: 1. Short-term (~1 RTT) buffering: this would be handy for segmentation-reassembly and retransmit loop use cases. Basically some “buffer”, and “remove” actions would be needed preferably with timer support 2. More generic buffering and time-based events: for programmable traffic management, packet scheduling or keepalive messages we could use a more generic method. Possibly the problem is very similar to the previous on the API level, but in these cases there is probably less limitations on the buffer size, which could make this a bit more tricky from hardware development perspective 3. Modular pipelines: this would mean that we could specify multiple pipelines and a “master” pipeline that would call the underlying ones as subroutines. Perhaps more than 2 layers could be allowed, e.g. a “master” can also be re-used as a module by a higher layer pipeline. Probably this could be solved entirely on the compiler level. We also described a few less important cases (e.g. re-using tables, registers, etc.), but we could easily find workarounds to overcome those limitations, so let’s focus on the ones above. I’m quite new to this community although we’ve been using P4 for a while now. So I don’t really know what is the best way to start discussing these issues and if you find these useful, how to start working on (some of) these. Thanks for any hints and help. BR, Gergely _______________________________________________ P4-design mailing list -- p4-des...@lists.p4.org <mailto:p4-des...@lists.p4.org> To unsubscribe send an email to p4-design-le...@lists.p4.org <mailto:p4-design-le...@lists.p4.org> -- You received this message because you are subscribed to the Google Groups "p4-design" group. To unsubscribe from this group and stop receiving emails from it, send an email to p4-design+unsubscr...@lists.p4.org <mailto:p4-design+unsubscr...@lists.p4.org> . To view this discussion on the web visit https://groups.google.com/a/lists.p4.org/d/msgid/p4-design/0c1501d8abf6%246491a240%242db4e6c0%24%40mnkcg.com <https://groups.google.com/a/lists.p4.org/d/msgid/p4-design/0c1501d8abf6%246491a240%242db4e6c0%24%40mnkcg.com?utm_medium=email&utm_source=footer> .
smime.p7s
Description: S/MIME cryptographic signature
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21783): https://lists.fd.io/g/vpp-dev/message/21783 Mute This Topic: https://lists.fd.io/mt/92919589/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
