On Mon, 2 Oct 2023 at 20:21, Jeff Behrns via NANOG <nanog@nanog.org> wrote:
> Encountered an issue with an MX204 using all 4x100G ports and a logical > tunnel to hairpin a VRF. The tunnel started dropping packets around 8Gbps. > I bumped up tunnel-services BW from 10G to 100G which made the problem > worse; the tunnel was now limited to around 1.3Gbps. To my knowledge with > Trio PFE you shouldn't have to disable a physical port to allocate bandwidth > for tunnel-services. Any helpful info is appreciated. You might have more luck in j-nsp. But yes you don't need any physical interface in trio to do tunneling. I can't explain your problem, and you probably need JTAC help. I would appreciate it if you'd circle back and tell what the problem was. How it works is that when PPE decides it needs to tunnel the packet, you're going to send the packet back to MQ via SERDES (which will then send it again to some PPE, not the same). I think what that bandwidth command does is change the stream allocation, you should see it in 'show <MQ/XM...> <#> stream'. In theory, because PPE can process packet forever (well, until watchdog kills the PPE for thinking it is stuck) you could very cheaply do outer+inner at the local PPE, but I think that would mean that certain features like QoS would not work on the inner interface, so I think all this expensive recirculation and SERDES consumption is to satisfy quite limited need, and it should be possible to implement some 'performance mode' for tunneling, where these MQ/XM provided features are not available, but performance cost in most cases is negligible. In parallel to opening the JTAC case, you might want to try to experiment in which FPC/PIC you set the tunneling bandwidth to. I don't understand how the tunneling would work if the MQ/XM is remote, like would you then also steal fabric capacity every time you tunnel, not just MQ>LU>MQ>LU SERDES, but MQ>LU>MQ>FAB>MQ>LU. So intuitively I would recommend ensuring you have the bandwidth configured at the local PFE, if you don't know what the local PFE is, just configure it everywhere? Also you could consult several counters to see if some stream or fabric is congested, and these tunneled packets are being sent over congested fabric every time with lower fabric qos. I don't understand why the bandwidth command is a thing, and why you can configure where it is. To me it seems obvious they should always handle tunneling strictly locally, never over fabric, because you always end up stealing more capacity if you send it to remote MQ. That is, implicitly it should be on for every MQ, and every PPE tunnel via local MQ. -- ++ytti
