Re: Netmap Library not getting installed on custom kernel installation

[Olivier Cochard-Labbé]

On Wed, Dec 16, 2020 at 9:53 PM Vincenzo Maffione 
wrote:

>
> On a side note, the netmap tools (pkt-gen, bridge, lb, etc.) should really
> be a port. Another TODO item.
>
>
There is already one port for an old version of pkt-gen:
https://svnweb.freebsd.org/ports/head/net/pkt-gen/

And here is a custom port's patch to upgrade this port to a not-so-old
version including a quick range bug fix (cleaner version
fresh-from-today official netmap github), and adding a new option to
prevent doing software IP & UDP checksum by default (because it consumes a
lot of ressource on 40G or 100G NIC and Chelsio NIC are able to do hardware
checksum in netmap mode):
https://github.com/ocochard/BSDRP/blob/master/BSDRP/patches/ports.pkt-gen.patch

Regards,

Olivier
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap Library not getting installed on custom kernel installation

[Rajesh Kumar]

Hi,

Got around this issue by manually copying necessary in /usr/obj

Looks like libnetmap (src/lib/libnetmap) is not built and installed by
default. Manually built them and copied the "libnetmap.h" and
"libnetmap.so" (not just the header file) to appropriate directories in
/usr/obj helps to get past the issue.  But not sure why libnetmap is not
built and installed though "device netmap" is set in the config file.

Manually copying may not be the right approach. Can anyone suggest a
cleaner way of getting things done?

Thanks,
Rajesh.

On Wed, Dec 16, 2020 at 3:33 PM Rajesh Kumar  wrote:

> Hi,
>
> I am trying to compile the netmap tools(pkt-gen, bridge etc.,) and getting
> the below error.
>
>
>
>
>
> */root//freebsd/tools/tools/netmap/pkt-gen.c:47:10: fatal error:
> 'libnetmap.h' file not found#include  ^1
> error generated.*** Error code 1*
>
> On debug, I don't see the libnetmap.h file getting installed in the
> /usr/obj/ directory.  Whereas, in another similar machine I have the file
> in /usr/obj and compilation of netmap tools goes fine.  For the test, I
> just copied libnetmap.h from the source and it leads to linker error. So
> seems the libnetmap library is not installed properly.
>
> I installed a custom kernel just disabling the debug options with the
> FreeBSD-CURRENT branch.  After rebooting to the custom kernel, I tried to
> compile the netmap tools (with some changes) and ran into the above error.
>
> How to get the netmap tools compiled in this scenario? Am i missing
> something?
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap bridge not working with 10G Ethernet ports

[Rajesh Kumar]

Hi Vincenzo,


On Tue, Nov 24, 2020 at 8:54 PM Vincenzo Maffione 
wrote:

>
> Yeah, it's weird because axgbe also uses iflib(4). If the driver exposes
> NIC head/tail pointers (sysctl) it may be useful to check what happens
> there.
> It may be that the NIC is dropping these packets for some reason.
>

Looks like, "ifdi_promisc_set" routine of the driver is not getting
triggered properly.  Forcibly setting the promisc mode from driver is
solving the packet drop issue. Now I see the ARP reply packet.

axgbe has split header support, which causes trouble starting from ping
packets due to incompatibility with Iflib/Netmap and associated utilities.
I made some changes to the driver, Iflib and Netmap utilities and having
the netmap bridge working with axgbe now.

I am working on a clean fix for promisc mode setting and incompatibility
issue. Will submit the changes for review once done.

Thanks for your inputs and support.

Done in r367920.
>

Thank you.

Thanks,
Rajesh.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap bridge not working with 10G Ethernet ports

[Rajesh Kumar]

Hi Vincenzo,

Thanks for pointing this.

On Sat, Nov 21, 2020 at 10:40 PM Vincenzo Maffione 
wrote:

> # ifconfig ix0 promisc
> # ifconfig ix1 promisc
>
> This is an additional requirement when using netmap bridge, because that
> is not done automatically (differently from what happens with if_bridge(4)).
> If promisc is not enabled, the NIC will drop any unicast packet that is
> not directed to the NIC's address (e.g. the ARP reply in your case).
> Broadcast packets will of course pass (e.g. the ARP request). This explains
> the absence of IRQs and the head/tail pointers not being updated.
> So no bugs AFAIK.
>

Setting the interfaces in promiscous mode makes things to work properly.

I tried the same with AMD Ports and it's still not working.  I believe this
is something specific to if_axp driver. I will see what is going wrong with
packet forwarding with AMD ports. Thanks for pointing this out.

I figured it out the hard way, but it was actually also documented on the
> github (https://github.com/luigirizzo/netmap#receiver-does-not-receive).
> I will add it to the netmap bridge man page.
>

That would be helpful. Thanks.


> Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione <
> vmaffi...@freebsd.org> ha scritto:
>
>>
>>
>> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar 
>> ha scritto:
>>
>>> Hi Vincenzo,
>>>
>>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione 
>>> wrote:
>>>

 Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4)
 uses iflib(4). This means that actually if_axp(4) has native netmap
 support, because iflib(4) has native netmap support.


>>> It means that the driver has some modifications to allow netmap to
 directly program the NIC rings. These modifications are mostly the
 per-driver txsync and rxsyng routines.
 In case of iflib(4) drivers, these modifications are provided directly
 within the iflib(4) code, and therefore any driver using iflib will have
 native netmap support.

>>>
>>> Thanks for clarifying on the Native Netmap support.
>>>
>>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you
 are basically hitting the same issue of if_axp(4)
 At this point I must think that there is still some issue with the
 interaction between iflib(4) and netmap(4).

>>>
>>> Ok. Let me know if any more debug info needed in this part.
>>>
>>> I see. This info may be useful. Have you tried to look at interrupts
 (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing
 ARP replies)?

>>>
>>> It's interesting here. When I try with Intel NIC card. I see atleast 1
>>> interrupt raised.  But not sure whether that is for ARP reply. Because,
>>> when I try to dump the packet from "bridge"(modified) utility, I don't see
>>> any ARP reply packet getting dumped.
>>>
>>>
>>> *irq59: ix0:rxq01  0 (only 1 interrupt
>>> on the opposite side)*irq67: ix0:aq  2
>>>  0
>>>
>>> *irq68: ix1:rxq03  0  (you can see 3
>>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
>>>  2  0
>>>
>>> The same experiment, when I try with AMD inbuilt ports, I don't see that
>>> 1 interrupt also raised.
>>>
>>> irq81: ax0:dev_irq16  0
>>> irq83: ax0  2541  4
>>> irq93: ax1:dev_irq27  0
>>> irq95: ax1  2371  3
>>> *irq97: ax1:rxq03  0 (you can see 3
>>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from
>>> "ax0:rxq0" for ARP reply from System 2)*
>>>
>>> I will do some more testing to see whether this behavior is consistent
>>> or intermittent.
>>>
>>> Also the igb(4) driver is using iflib(4). So the involved netmap code is
 the same as ix(4) and if_axp(4).
 This is something that I'm not able to understand right now.
 It does not look like something related to offloads.

 Next week I will try to see if I can reproduce your issue with em(4),
 and report back. That's still an Intel driver using iflib(4).

>>>
>>> The "igb(4)" driver, with which things are working now is related to
>>> em(4) driver (may be for newer hardware version).  Initially we faced
>>> similar issue with igb(4) driver as well. After reverting the following
>>> commits, things started to work.  Thanks to Stephan Dewt (copied) for
>>> pointing this.  But it still fails with ix(4) driver and if_axp(4) driver.
>>>
>>>
>>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf
>>>
>>> Thanks for providing your inputs on this issue Vincenzo.  Let me know
>>> for any more details that you need.
>>>
>>>
>> I was able to reproduce your issue on FreeBSD-CURRENT running within a
>> QEMU VM, with two em(4) devices and the netmap bridge running between them.
>> I see the ARP request packet 

Re: Netmap bridge not working with 10G Ethernet ports

[Vincenzo Maffione]

Hi Rajesh,
  I think the issue here is simply that you have not enabled promiscuous
mode on your interfaces.
# ifconfig ix0 promisc
# ifconfig ix1 promisc

This is an additional requirement when using netmap bridge, because that is
not done automatically (differently from what happens with if_bridge(4)).
If promisc is not enabled, the NIC will drop any unicast packet that is not
directed to the NIC's address (e.g. the ARP reply in your case). Broadcast
packets will of course pass (e.g. the ARP request). This explains the
absence of IRQs and the head/tail pointers not being updated.
So no bugs AFAIK.

I figured it out the hard way, but it was actually also documented on the
github (https://github.com/luigirizzo/netmap#receiver-does-not-receive).
I will add it to the netmap bridge man page.

Cheers,
  Vincenzo


Il giorno sab 21 nov 2020 alle ore 17:06 Vincenzo Maffione <
vmaffi...@freebsd.org> ha scritto:

>
>
> Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar 
> ha scritto:
>
>> Hi Vincenzo,
>>
>> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione 
>> wrote:
>>
>>>
>>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses
>>> iflib(4). This means that actually if_axp(4) has native netmap support,
>>> because iflib(4) has native netmap support.
>>>
>>>
>> It means that the driver has some modifications to allow netmap to
>>> directly program the NIC rings. These modifications are mostly the
>>> per-driver txsync and rxsyng routines.
>>> In case of iflib(4) drivers, these modifications are provided directly
>>> within the iflib(4) code, and therefore any driver using iflib will have
>>> native netmap support.
>>>
>>
>> Thanks for clarifying on the Native Netmap support.
>>
>> Ok, this makes sense, because also ix(4) uses iflib, and therefore you
>>> are basically hitting the same issue of if_axp(4)
>>> At this point I must think that there is still some issue with the
>>> interaction between iflib(4) and netmap(4).
>>>
>>
>> Ok. Let me know if any more debug info needed in this part.
>>
>> I see. This info may be useful. Have you tried to look at interrupts
>>> (e.g. `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing
>>> ARP replies)?
>>>
>>
>> It's interesting here. When I try with Intel NIC card. I see atleast 1
>> interrupt raised.  But not sure whether that is for ARP reply. Because,
>> when I try to dump the packet from "bridge"(modified) utility, I don't see
>> any ARP reply packet getting dumped.
>>
>>
>> *irq59: ix0:rxq01  0 (only 1 interrupt on
>> the opposite side)*irq67: ix0:aq  2  0
>>
>> *irq68: ix1:rxq03  0  (you can see 3
>> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
>>2  0
>>
>> The same experiment, when I try with AMD inbuilt ports, I don't see that
>> 1 interrupt also raised.
>>
>> irq81: ax0:dev_irq16  0
>> irq83: ax0  2541  4
>> irq93: ax1:dev_irq27  0
>> irq95: ax1  2371  3
>> *irq97: ax1:rxq03  0 (you can see 3
>> interrupts for 3 ARP requests from System 1, but no interrupt is seen from
>> "ax0:rxq0" for ARP reply from System 2)*
>>
>> I will do some more testing to see whether this behavior is consistent or
>> intermittent.
>>
>> Also the igb(4) driver is using iflib(4). So the involved netmap code is
>>> the same as ix(4) and if_axp(4).
>>> This is something that I'm not able to understand right now.
>>> It does not look like something related to offloads.
>>>
>>> Next week I will try to see if I can reproduce your issue with em(4),
>>> and report back. That's still an Intel driver using iflib(4).
>>>
>>
>> The "igb(4)" driver, with which things are working now is related to
>> em(4) driver (may be for newer hardware version).  Initially we faced
>> similar issue with igb(4) driver as well. After reverting the following
>> commits, things started to work.  Thanks to Stephan Dewt (copied) for
>> pointing this.  But it still fails with ix(4) driver and if_axp(4) driver.
>>
>>
>> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf
>>
>> Thanks for providing your inputs on this issue Vincenzo.  Let me know for
>> any more details that you need.
>>
>>
> I was able to reproduce your issue on FreeBSD-CURRENT running within a
> QEMU VM, with two em(4) devices and the netmap bridge running between them.
> I see the ARP request packet received on em0 (with associated IRQ), and
> forwarded on em1. However, the ARP reply coming on em1 does not trigger an
> IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as
> they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird,
> and lets me think that the issue is more likely driver-related than
> netmap/iflib-related.
> In any case, would you mind filing the issue 

Re: Netmap bridge not working with 10G Ethernet ports

[Vincenzo Maffione]

Il giorno ven 20 nov 2020 alle ore 14:31 Rajesh Kumar 
ha scritto:

> Hi Vincenzo,
>
> On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione 
> wrote:
>
>>
>> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses
>> iflib(4). This means that actually if_axp(4) has native netmap support,
>> because iflib(4) has native netmap support.
>>
>>
> It means that the driver has some modifications to allow netmap to
>> directly program the NIC rings. These modifications are mostly the
>> per-driver txsync and rxsyng routines.
>> In case of iflib(4) drivers, these modifications are provided directly
>> within the iflib(4) code, and therefore any driver using iflib will have
>> native netmap support.
>>
>
> Thanks for clarifying on the Native Netmap support.
>
> Ok, this makes sense, because also ix(4) uses iflib, and therefore you are
>> basically hitting the same issue of if_axp(4)
>> At this point I must think that there is still some issue with the
>> interaction between iflib(4) and netmap(4).
>>
>
> Ok. Let me know if any more debug info needed in this part.
>
> I see. This info may be useful. Have you tried to look at interrupts (e.g.
>> `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing ARP
>> replies)?
>>
>
> It's interesting here. When I try with Intel NIC card. I see atleast 1
> interrupt raised.  But not sure whether that is for ARP reply. Because,
> when I try to dump the packet from "bridge"(modified) utility, I don't see
> any ARP reply packet getting dumped.
>
>
> *irq59: ix0:rxq01  0 (only 1 interrupt on
> the opposite side)*irq67: ix0:aq  2  0
>
> *irq68: ix1:rxq03  0  (you can see 3
> interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
>2  0
>
> The same experiment, when I try with AMD inbuilt ports, I don't see that 1
> interrupt also raised.
>
> irq81: ax0:dev_irq16  0
> irq83: ax0  2541  4
> irq93: ax1:dev_irq27  0
> irq95: ax1  2371  3
> *irq97: ax1:rxq03  0 (you can see 3
> interrupts for 3 ARP requests from System 1, but no interrupt is seen from
> "ax0:rxq0" for ARP reply from System 2)*
>
> I will do some more testing to see whether this behavior is consistent or
> intermittent.
>
> Also the igb(4) driver is using iflib(4). So the involved netmap code is
>> the same as ix(4) and if_axp(4).
>> This is something that I'm not able to understand right now.
>> It does not look like something related to offloads.
>>
>> Next week I will try to see if I can reproduce your issue with em(4), and
>> report back. That's still an Intel driver using iflib(4).
>>
>
> The "igb(4)" driver, with which things are working now is related to em(4)
> driver (may be for newer hardware version).  Initially we faced similar
> issue with igb(4) driver as well. After reverting the following commits,
> things started to work.  Thanks to Stephan Dewt (copied) for pointing
> this.  But it still fails with ix(4) driver and if_axp(4) driver.
>
>
> https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf
>
> Thanks for providing your inputs on this issue Vincenzo.  Let me know for
> any more details that you need.
>
>
I was able to reproduce your issue on FreeBSD-CURRENT running within a QEMU
VM, with two em(4) devices and the netmap bridge running between them.
I see the ARP request packet received on em0 (with associated IRQ), and
forwarded on em1. However, the ARP reply coming on em1 does not trigger an
IRQ on em1, and indeed the NIC RX head/tail pointers are not incremented as
they should (`sysctl -a | grep em.1 | grep queue_rx`) ... that is weird,
and lets me think that the issue is more likely driver-related than
netmap/iflib-related.
In any case, would you mind filing the issue on the bugzilla, so that we
can properly track this issue?

Thanks,
  Vincenzo


> Thanks,
> Rajesh.
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap bridge not working with 10G Ethernet ports

[Rajesh Kumar]

Hi Vincenzo,

On Fri, Nov 20, 2020 at 3:20 AM Vincenzo Maffione 
wrote:

>
> Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses
> iflib(4). This means that actually if_axp(4) has native netmap support,
> because iflib(4) has native netmap support.
>
>
It means that the driver has some modifications to allow netmap to directly
> program the NIC rings. These modifications are mostly the per-driver txsync
> and rxsyng routines.
> In case of iflib(4) drivers, these modifications are provided directly
> within the iflib(4) code, and therefore any driver using iflib will have
> native netmap support.
>

Thanks for clarifying on the Native Netmap support.

Ok, this makes sense, because also ix(4) uses iflib, and therefore you are
> basically hitting the same issue of if_axp(4)
> At this point I must think that there is still some issue with the
> interaction between iflib(4) and netmap(4).
>

Ok. Let me know if any more debug info needed in this part.

I see. This info may be useful. Have you tried to look at interrupts (e.g.
> `vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing ARP
> replies)?
>

It's interesting here. When I try with Intel NIC card. I see atleast 1
interrupt raised.  But not sure whether that is for ARP reply. Because,
when I try to dump the packet from "bridge"(modified) utility, I don't see
any ARP reply packet getting dumped.


*irq59: ix0:rxq01  0 (only 1 interrupt on
the opposite side)*irq67: ix0:aq  2  0

*irq68: ix1:rxq03  0  (you can see 3
interrupts for 3 ARP requests from System 1)*irq76: ix1:aq
 2  0

The same experiment, when I try with AMD inbuilt ports, I don't see that 1
interrupt also raised.

irq81: ax0:dev_irq16  0
irq83: ax0  2541  4
irq93: ax1:dev_irq27  0
irq95: ax1  2371  3
*irq97: ax1:rxq03  0 (you can see 3
interrupts for 3 ARP requests from System 1, but no interrupt is seen from
"ax0:rxq0" for ARP reply from System 2)*

I will do some more testing to see whether this behavior is consistent or
intermittent.

Also the igb(4) driver is using iflib(4). So the involved netmap code is
> the same as ix(4) and if_axp(4).
> This is something that I'm not able to understand right now.
> It does not look like something related to offloads.
>
> Next week I will try to see if I can reproduce your issue with em(4), and
> report back. That's still an Intel driver using iflib(4).
>

The "igb(4)" driver, with which things are working now is related to em(4)
driver (may be for newer hardware version).  Initially we faced similar
issue with igb(4) driver as well. After reverting the following commits,
things started to work.  Thanks to Stephan Dewt (copied) for pointing
this.  But it still fails with ix(4) driver and if_axp(4) driver.

https://github.com/freebsd/freebsd/commit/e12efc2c9e434075d0740e2e2e9e2fca2ad5f7cf

Thanks for providing your inputs on this issue Vincenzo.  Let me know for
any more details that you need.

Thanks,
Rajesh.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap bridge not working with 10G Ethernet ports

[Vincenzo Maffione]

Il giorno gio 19 nov 2020 alle ore 12:28 Rajesh Kumar 
ha scritto:

> Hi Vincenzo,
>
> Thanks for your reply.
>
> On Thu, Nov 19, 2020 at 3:16 AM Vincenzo Maffione 
> wrote:
>
>>
>> This looks like if_axe(4) driver, and therefore there's no native netmap
>> support, which means you are falling back on
>> the emulated netmap adapter. Are these USB dongles? If so, how can they
>> be 10G?
>>
>
> The Driver I am working with is "if_axp" (sys/dev/axgbe).  This is AMD
> 10Gigabit Ethernet Driver. This is recently committed upstream. Yes, it
> doesn't have a Native netmap support, but uses the netmap stack which is
> existing already.  These are inbuilt SFP ports with our test board and not
> USB dongles.
>

Ok, now it makes sense. Thanks for clarifying. I see that if_axp(4) uses
iflib(4). This means that actually if_axp(4) has native netmap support,
because iflib(4) has native netmap support.


> Does Native netmap mean the hardware capability which needs to be
> programmed appropriately from driver side?  Any generic documentation
> regarding the same?
>

It means that the driver has some modifications to allow netmap to directly
program the NIC rings. These modifications are mostly the per-driver txsync
and rxsyng routines.
In case of iflib(4) drivers, these modifications are provided directly
within the iflib(4) code, and therefore any driver using iflib will have
native netmap support.


>
>> In this kind of configuration it is mandatory to disable all the NIC
>> offloads, because netmap does not program the NIC
>> to honor them, e.g.:
>>
>> # ifconfig ax0 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
>> # ifconfig ax1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
>>
>
> Earlier, I haven't tried disabling the Offload capabilities.  But I tried
> now, but it still behaves the same way.  ARP replies doesn't seem to reach
> the bridge (or dropped) to be forwarded.  I will collect the details for
> AMD driver. Tried the same test with another 10G card (Intel "ix" driver)
> also exhibits similar behavior.  Details below.
>

Ok, this makes sense, because also ix(4) uses iflib, and therefore you are
basically hitting the same issue of if_axp(4)
At this point I must think that there is still some issue with the
interaction between iflib(4) and netmap(4).


>
>
>> a) I tried with another vendor 10G NIC card. It behaves the same way. So
>>> this issue doesn't seem to be generic and not hardware specific.
>>>
>>
>> Which driver are those NICs using? That makes the difference. I guess
>> it's still a driver with no native netmap support, hence
>> you are using the same emulated adapter
>>
>
> I am using the "ix" driver (Intel 10G NIC adapter).  I guess this driver
> also doesn't support Native Netmap.  Please correct me if I am wrong.  I
> tried disabling the offload capabilities with this device/driver and tested
> and still observed the netmap bridging fails.
>

As I stated above, ix(4) has netmap support, like any iflib(4) driver.


> root@fbsd_cur# sysctl dev.ix.0 | grep tx_packets
> dev.ix.0.queue7.tx_packets: 0
> dev.ix.0.queue6.tx_packets: 0
> dev.ix.0.queue5.tx_packets: 0
> dev.ix.0.queue4.tx_packets: 0
> dev.ix.0.queue3.tx_packets: 0
> dev.ix.0.queue2.tx_packets: 0
> dev.ix.0.queue1.tx_packets: 0
> *dev.ix.0.queue0.tx_packets: 3*
> root@fbsd_cur# sysctl dev.ix.0 | grep rx_packets
> dev.ix.0.queue7.rx_packets: 0
> dev.ix.0.queue6.rx_packets: 0
> dev.ix.0.queue5.rx_packets: 0
> dev.ix.0.queue4.rx_packets: 0
> dev.ix.0.queue3.rx_packets: 0
> dev.ix.0.queue2.rx_packets: 0
> dev.ix.0.queue1.rx_packets: 0
> dev.ix.0.queue0.rx_packets: 0
> root@fbsd_cur # sysctl dev.ix.1 | grep tx_packets
> dev.ix.1.queue7.tx_packets: 0
> dev.ix.1.queue6.tx_packets: 0
> dev.ix.1.queue5.tx_packets: 0
> dev.ix.1.queue4.tx_packets: 0
> dev.ix.1.queue3.tx_packets: 0
> dev.ix.1.queue2.tx_packets: 0
> dev.ix.1.queue1.tx_packets: 0
> dev.ix.1.queue0.tx_packets: 0
> root@fbsd_cur # sysctl dev.ix.1 | grep rx_packets
> dev.ix.1.queue7.rx_packets: 0
> dev.ix.1.queue6.rx_packets: 0
> dev.ix.1.queue5.rx_packets: 0
> dev.ix.1.queue4.rx_packets: 0
> dev.ix.1.queue3.rx_packets: 0
> dev.ix.1.queue2.rx_packets: 0
> dev.ix.1.queue1.rx_packets: 0
>
> *dev.ix.1.queue0.rx_packets: 3*
>
> You can see "ix1" received 3 packets (ARP requests) from system 1 and
> transmitted 3 packets to system 2 via "ix0". But ARP reply from system 2 is
> not captured or forwared properly.
>

I see. This info may be useful. Have you tried to look at interrupts (e.g.
`vmstat -i`), to see if "ix0" gets any RX interrupts (for the missing ARP
replies)?


>
> You can see the checksum features disabled (except VLAN_HWCSIM) on both
> interfaces.  And you can see both interfaces active and Link up.
>
> root@fbsd_cur # ifconfig -a
> ix0: flags=8862 metric 0 mtu 1500
>
> options=48538b8
> ether a0:36:9f:a5:49:90
> media: Ethernet autoselect (100baseTX )
> status: active
> nd6 options=29
>
> ix1: flags=8862 metric 0 mtu 1500
>
> options=48538b8

Re: Netmap bridge not working with 10G Ethernet ports

[Rajesh Kumar]

Hi Vincenzo,

Thanks for your reply.

On Thu, Nov 19, 2020 at 3:16 AM Vincenzo Maffione 
wrote:

>
> This looks like if_axe(4) driver, and therefore there's no native netmap
> support, which means you are falling back on
> the emulated netmap adapter. Are these USB dongles? If so, how can they be
> 10G?
>

The Driver I am working with is "if_axp" (sys/dev/axgbe).  This is AMD
10Gigabit Ethernet Driver. This is recently committed upstream. Yes, it
doesn't have a Native netmap support, but uses the netmap stack which is
existing already.  These are inbuilt SFP ports with our test board and not
USB dongles.

Does Native netmap mean the hardware capability which needs to be
programmed appropriately from driver side?  Any generic documentation
regarding the same?


> In this kind of configuration it is mandatory to disable all the NIC
> offloads, because netmap does not program the NIC
> to honor them, e.g.:
>
> # ifconfig ax0 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
> # ifconfig ax1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
>

Earlier, I haven't tried disabling the Offload capabilities.  But I tried
now, but it still behaves the same way.  ARP replies doesn't seem to reach
the bridge (or dropped) to be forwarded.  I will collect the details for
AMD driver. Tried the same test with another 10G card (Intel "ix" driver)
also exhibits similar behavior.  Details below.


> a) I tried with another vendor 10G NIC card. It behaves the same way. So
>> this issue doesn't seem to be generic and not hardware specific.
>>
>
> Which driver are those NICs using? That makes the difference. I guess it's
> still a driver with no native netmap support, hence
> you are using the same emulated adapter
>

I am using the "ix" driver (Intel 10G NIC adapter).  I guess this driver
also doesn't support Native Netmap.  Please correct me if I am wrong.  I
tried disabling the offload capabilities with this device/driver and tested
and still observed the netmap bridging fails.

root@fbsd_cur# sysctl dev.ix.0 | grep tx_packets
dev.ix.0.queue7.tx_packets: 0
dev.ix.0.queue6.tx_packets: 0
dev.ix.0.queue5.tx_packets: 0
dev.ix.0.queue4.tx_packets: 0
dev.ix.0.queue3.tx_packets: 0
dev.ix.0.queue2.tx_packets: 0
dev.ix.0.queue1.tx_packets: 0
*dev.ix.0.queue0.tx_packets: 3*
root@fbsd_cur# sysctl dev.ix.0 | grep rx_packets
dev.ix.0.queue7.rx_packets: 0
dev.ix.0.queue6.rx_packets: 0
dev.ix.0.queue5.rx_packets: 0
dev.ix.0.queue4.rx_packets: 0
dev.ix.0.queue3.rx_packets: 0
dev.ix.0.queue2.rx_packets: 0
dev.ix.0.queue1.rx_packets: 0
dev.ix.0.queue0.rx_packets: 0
root@fbsd_cur # sysctl dev.ix.1 | grep tx_packets
dev.ix.1.queue7.tx_packets: 0
dev.ix.1.queue6.tx_packets: 0
dev.ix.1.queue5.tx_packets: 0
dev.ix.1.queue4.tx_packets: 0
dev.ix.1.queue3.tx_packets: 0
dev.ix.1.queue2.tx_packets: 0
dev.ix.1.queue1.tx_packets: 0
dev.ix.1.queue0.tx_packets: 0
root@fbsd_cur # sysctl dev.ix.1 | grep rx_packets
dev.ix.1.queue7.rx_packets: 0
dev.ix.1.queue6.rx_packets: 0
dev.ix.1.queue5.rx_packets: 0
dev.ix.1.queue4.rx_packets: 0
dev.ix.1.queue3.rx_packets: 0
dev.ix.1.queue2.rx_packets: 0
dev.ix.1.queue1.rx_packets: 0

*dev.ix.1.queue0.rx_packets: 3*

You can see "ix1" received 3 packets (ARP requests) from system 1 and
transmitted 3 packets to system 2 via "ix0". But ARP reply from system 2 is
not captured or forwared properly.

You can see the checksum features disabled (except VLAN_HWCSIM) on both
interfaces.  And you can see both interfaces active and Link up.

root@fbsd_cur # ifconfig -a
ix0: flags=8862 metric 0 mtu 1500

options=48538b8
ether a0:36:9f:a5:49:90
media: Ethernet autoselect (100baseTX )
status: active
nd6 options=29

ix1: flags=8862 metric 0 mtu 1500

options=48538b8
ether a0:36:9f:a5:49:92
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29

>
> b) Trying with another vendor 1G NIC card, things are working.  So not
>> sure, what makes a difference here.  The ports in System 1 and System 2
>> are
>> USB attached Ethernet capable of maximum speed of 1G.  So does connecting
>> 1G to 10G bridge ports is having any impact?
>>
>
> I don't think so. On each p2p link the NICs will negotiate 1G speed.
> In any case, what driver was this one?
>

This is "igb" driver. Intel 1G NIC Card.

Thanks,
Rajesh.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap bridge not working with 10G Ethernet ports

[Vincenzo Maffione]

Hi,

Il giorno mer 18 nov 2020 alle ore 08:13 Rajesh Kumar 
ha scritto:

> Hi,
>
> I am testing a 10G Network driver with Netmap "bridge" utility, where it
> doesn't seem to work. Here is my setup details.
>
> *System under Test:*  Running FreeBSD CURRENT.  Has two inbuilt 10G NIC
> ports.
> *System 1:* Running Ubuntu, whose network port is connected to Port1 of
> System Under Test
> *System 2:* Running FreeBSD CURRENT, whose network port is connected to
> Port 0 of System Under Test.
>
> Bridged the Port0 and Port1 of System Under Test using the Netmap "bridge"
> utility. Able to see interfaces coming up active and Link UP.
> # bridge -c -v -i netmap:ax0 -i netmap:ax1
>
> This looks like if_axe(4) driver, and therefore there's no native netmap
support, which means you are falling back on
the emulated netmap adapter. Are these USB dongles? If so, how can they be
10G?


> Then tried pinging from System 1 to System 2. It fails.
>
> *Observations:*
> 1. ARP request from System 1 goes to bridge port 1 (netmap_rxsync) and then
> forwarded to port 0 (netmap_txsync)
> 2. ARP request is received in System 2 (via bridge port 0) and ARP reply is
> being sent from System 2.
> 3. ARP reply from System 2 seems to be not reaching bridge port 0 to get
> forwarded to bridge 1 and hence to System 1.
> 4. Above 3 steps happen 3 times for ARP resolution cycle and then fails.
> Hence the ping fails.
>
> On Debugging, when the ARP reply is being sent from System 2, I don't see
> any interrupt triggered on the bridge port 0 in system under test.
>
> In this kind of configuration it is mandatory to disable all the NIC
offloads, because netmap does not program the NIC
to honor them, e.g.:

# ifconfig ax0 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
# ifconfig ax1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6


> Netstat in system under test, doesn't show any receive or drop counters
> incremented. But as I understand netstat capture the stats above the netmap
> stack. Hence not reflecting the counts.
>

Correct.


>
> *Note:*
> a) I tried with another vendor 10G NIC card. It behaves the same way. So
> this issue doesn't seem to be generic and not hardware specific.
>

Which driver are those NICs using? That makes the difference. I guess it's
still a driver with no native netmap support, hence
you are using the same emulated adapter.


> b) Trying with another vendor 1G NIC card, things are working.  So not
> sure, what makes a difference here.  The ports in System 1 and System 2 are
> USB attached Ethernet capable of maximum speed of 1G.  So does connecting
> 1G to 10G bridge ports is having any impact?
>

I don't think so. On each p2p link the NICs will negotiate 1G speed.
In any case, what driver was this one?


> c) We have verified the same 10G driver with pkt-gen utility and things are
> working. Facing issue only when using "bridge" utility.
>

That may be because pkt-gen does not care about checksums, whereas the
TCP/IP stack does.
Hence the need to disable offloads (see above).

Cheers,
  Vincenzo


> So, wondering how the ARP reply packet is getting lost here. Any ideas to
> debug?
>
> Thanks,
> Rajesh.
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap - Vale switch - tcp problem

[Vincenzo Maffione]

Il giorno sab 6 giu 2020 alle ore 20:32 Anthony Arnaud <
antho.arnaudi...@gmail.com> ha scritto:

> This works!!
> A good news!
> It works even if I connect the nic to vale switch directly from ssh, the
> connection does not drop like before your patch.
>

Yes, connecting the vtnet0 to the vale switch it's what I meant.


> Can you list the steps to setup the bridge in your tests? I thing
> "ifconfig bridge create" etc.
>

I was doing my tests on a Linux host (KVM hypervisor), but it should be the
same if you run on FreeBSD.
The steps are:

# ifconfig bridge0 create
# ifconfig bridge0 up 192.168.1.1/24
# ifconfig tap0 create
# ifconfig tap0 up
# ifconfig bridge0 addm tap0

And then run your VM in such a way that your vtnet0 is backed by tap0.

It's basically the same instructions you find here:
https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html

Now I have to understand why it doesn't work in my previous case.
>

At this point I think if you still don't see TCP traffic you are likely to
have some misconfiguration, rather than a VALE/vtnet issue.

It may be useful to know that netmap has an intercept mechanism, "netmap
monitor ports", that you can use to sniff on any netmap port, while the
netmap port is being used by another netmap application.

For instance, if you run
# pkt-gen -XX -i netmap:vtnet0/r
you will sniff all the packets that are being received on the vtnet0 port.
Otherwise you can sniff packets being transmitted:
# pkt-gen -XX -i netmap:vtnet0/t
The same goes for VALE ports:
# pkt-gen -XX -i vale0:1/r

This is very similar to tcpdump (although pkt-gen does not do packet decode
and pretty printing).

Cheers,
  VIncenzo


> Thanks,
> Antho
>
> Il giorno sab 6 giu 2020 alle ore 09:52 Vincenzo Maffione <
> vmaffi...@freebsd.org> ha scritto:
>
>>
>>
>> Il giorno ven 5 giu 2020 alle ore 18:16 Anthony Arnaud <
>> antho.arnaudi...@gmail.com> ha scritto:
>>
>>> Hi Vincenzo,
>>> Thank you for your answers.
>>> I upgraded a guest machine in proxmox env to FreeBSD 13.0-CURRENT #0
>>> r361830.
>>> After that I compiled the tools as usual from /src/tools/tools/netmap
>>> I configured 2 NIC vtnet
>>>
>>> vtnet0 with an ip 192.168.1.x
>>> vtnet1 without ip and csum disabled as mirror of vtnet0 (using Open
>>> vSwitch)
>>> Tcp traffic is generated by an ssh connection from my host to guest
>>> machine.
>>>
>>
>> Not clear where you run OVS. In the host or in the guest?
>>
>>
>>> "tcpdump -i vtnet1 tcp": at each keypress in ssh shell connected to
>>> 192.168.1.x I see few tcp packet sniffed in guest machine.
>>> But if I attach vtnet1 to vale switch, tcpdump no longer works as
>>> before. No tcp traffic is shown.
>>> Can I perform any other test about this?
>>>
>>
>> First, have you tried to leave vtnet1 alone and try:
>>
>> # ifconfig vtnet0 up 192.168.1.x -txcsum -rxcsum -tso4 -tso6
>> # vale-ctl -h vale0:vtnet0
>> and verified that you can ssh into 192.168.1.x from the host?
>>
>> At least you verify that the vale-ctl -h works also on your enivironment.
>>
>> Cheers,
>>   Vincenzo
>>
>>
>>> Cheers,
>>> Antho
>>>
>>> Il giorno mer 3 giu 2020 alle ore 19:53 Vincenzo Maffione <
>>> vmaffi...@freebsd.org> ha scritto:
>>>
 Hi Anthony,
   I fixed more bugs in the vtnet driver (in FreeBSD-CURRENT). As of
 r361760, I'm now able to run the following steps in a VM with a vtnet 
 device

 # ifconfig vtnet0 -txcsum -rxcsum -tso4 192.168.100.2/24
 # vale-ctl -h vale:vtnet0
 # netperf -H 192.168.100.1  # Run a netperf TCP_STREAM test to the host
 bridge interface (br0)

 Since TCP works correctly at reasonable speed I'm confident that most
 of the existing problems have gone.
 Let me know if you have any questions about this.

 Cheers,
   Vincenzo

 Il giorno lun 1 giu 2020 alle ore 18:22 Vincenzo Maffione <
 vmaffi...@freebsd.org> ha scritto:

> Hi Anthony,
>   I think there is more than a bug, drivers-related, that show up when
> you attach the interface to a vale switch.
> I've found and fixed some related to if_vtnet (see below). In any
> case, in my tests there is no difference between TCP traffic and other
> (UDP, ICMP, STP,...).
> The issues are not related to LRO, as I thought.
> There are still more bugs in vtnet and I'm trying to chase them.
> In the meanwhile it would help if you apply the patches below and try
> again with vtnet to see if the situation improves. They apply cleanly to
> 12.1 release.
>
> Regarding your problem with em devices, it is probably yet a different
> issue. It may be related to the iflib transition or not. It would help to
> try the same setup on stable/11 (which does not have iflib). I don't have
> an em device, but I will try with an emulated em in QEMU/KVM.
>
> Cheers,
>   Vincenzo
>
>
> 
> r361698 | vmaffione | 2020-06-01 

Re: Netmap - Vale switch - tcp problem

[Anthony Arnaud]

This works!!
A good news!
It works even if I connect the nic to vale switch directly from ssh, the
connection does not drop like before your patch.
Can you list the steps to setup the bridge in your tests? I thing "ifconfig
bridge create" etc.
Now I have to understand why it doesn't work in my previous case.

Thanks,
Antho

Il giorno sab 6 giu 2020 alle ore 09:52 Vincenzo Maffione <
vmaffi...@freebsd.org> ha scritto:

>
>
> Il giorno ven 5 giu 2020 alle ore 18:16 Anthony Arnaud <
> antho.arnaudi...@gmail.com> ha scritto:
>
>> Hi Vincenzo,
>> Thank you for your answers.
>> I upgraded a guest machine in proxmox env to FreeBSD 13.0-CURRENT #0
>> r361830.
>> After that I compiled the tools as usual from /src/tools/tools/netmap
>> I configured 2 NIC vtnet
>>
>> vtnet0 with an ip 192.168.1.x
>> vtnet1 without ip and csum disabled as mirror of vtnet0 (using Open
>> vSwitch)
>> Tcp traffic is generated by an ssh connection from my host to guest
>> machine.
>>
>
> Not clear where you run OVS. In the host or in the guest?
>
>
>> "tcpdump -i vtnet1 tcp": at each keypress in ssh shell connected to
>> 192.168.1.x I see few tcp packet sniffed in guest machine.
>> But if I attach vtnet1 to vale switch, tcpdump no longer works as before.
>> No tcp traffic is shown.
>> Can I perform any other test about this?
>>
>
> First, have you tried to leave vtnet1 alone and try:
>
> # ifconfig vtnet0 up 192.168.1.x -txcsum -rxcsum -tso4 -tso6
> # vale-ctl -h vale0:vtnet0
> and verified that you can ssh into 192.168.1.x from the host?
>
> At least you verify that the vale-ctl -h works also on your enivironment.
>
> Cheers,
>   Vincenzo
>
>
>> Cheers,
>> Antho
>>
>> Il giorno mer 3 giu 2020 alle ore 19:53 Vincenzo Maffione <
>> vmaffi...@freebsd.org> ha scritto:
>>
>>> Hi Anthony,
>>>   I fixed more bugs in the vtnet driver (in FreeBSD-CURRENT). As of
>>> r361760, I'm now able to run the following steps in a VM with a vtnet device
>>>
>>> # ifconfig vtnet0 -txcsum -rxcsum -tso4 192.168.100.2/24
>>> # vale-ctl -h vale:vtnet0
>>> # netperf -H 192.168.100.1  # Run a netperf TCP_STREAM test to the host
>>> bridge interface (br0)
>>>
>>> Since TCP works correctly at reasonable speed I'm confident that most of
>>> the existing problems have gone.
>>> Let me know if you have any questions about this.
>>>
>>> Cheers,
>>>   Vincenzo
>>>
>>> Il giorno lun 1 giu 2020 alle ore 18:22 Vincenzo Maffione <
>>> vmaffi...@freebsd.org> ha scritto:
>>>
 Hi Anthony,
   I think there is more than a bug, drivers-related, that show up when
 you attach the interface to a vale switch.
 I've found and fixed some related to if_vtnet (see below). In any case,
 in my tests there is no difference between TCP traffic and other (UDP,
 ICMP, STP,...).
 The issues are not related to LRO, as I thought.
 There are still more bugs in vtnet and I'm trying to chase them.
 In the meanwhile it would help if you apply the patches below and try
 again with vtnet to see if the situation improves. They apply cleanly to
 12.1 release.

 Regarding your problem with em devices, it is probably yet a different
 issue. It may be related to the iflib transition or not. It would help to
 try the same setup on stable/11 (which does not have iflib). I don't have
 an em device, but I will try with an emulated em in QEMU/KVM.

 Cheers,
   Vincenzo

 
 r361698 | vmaffione | 2020-06-01 16:14:29 + (Mon, 01 Jun 2020) | 8
 lines

 netmap: if_vtnet: avoid netmap ring wraparound

 netmap assumes the one "slot" is left unused to distinguish
 the empty ring and full ring conditions. This assumption was
 violated by vtnet_netmap_rxq_populate().

 MFC after:  1 week

 
 r361697 | vmaffione | 2020-06-01 16:12:09 + (Mon, 01 Jun 2020) | 8
 lines

 netmap: if_vtnet: replace vtnet_free_used()

 The functionality contained in this function is duplicated,
 as it is already available in vtnet_txq_free_mbufs()
 and vtnet_rxq_free_mbufs().

 MFC after:  1 week

 
 r361696 | vmaffione | 2020-06-01 16:10:44 + (Mon, 01 Jun 2020) | 13
 lines

 netmap: vtnet: fix RX virtqueue initialization bug

 The vtnet_netmap_rxq_populate() function erroneously assumed
 that kring->nr_hwcur = 0, i.e. the kring was in the initial
 state. However, this is not always the case: for example,
 when a vtnet reinit is triggered by some changes in the
 interface flags or capenable.
 This patch changes the behaviour of vtnet_netmap_kring_refill()
 so that it always starts publishing the netmap buffers starting
 from the current value of kring->nr_hwcur.

 MFC after:  1 

Re: Netmap - Vale switch - tcp problem

[Vincenzo Maffione]

Il giorno ven 5 giu 2020 alle ore 18:16 Anthony Arnaud <
antho.arnaudi...@gmail.com> ha scritto:

> Hi Vincenzo,
> Thank you for your answers.
> I upgraded a guest machine in proxmox env to FreeBSD 13.0-CURRENT #0
> r361830.
> After that I compiled the tools as usual from /src/tools/tools/netmap
> I configured 2 NIC vtnet
>
> vtnet0 with an ip 192.168.1.x
> vtnet1 without ip and csum disabled as mirror of vtnet0 (using Open
> vSwitch)
> Tcp traffic is generated by an ssh connection from my host to guest
> machine.
>

Not clear where you run OVS. In the host or in the guest?


> "tcpdump -i vtnet1 tcp": at each keypress in ssh shell connected to
> 192.168.1.x I see few tcp packet sniffed in guest machine.
> But if I attach vtnet1 to vale switch, tcpdump no longer works as before.
> No tcp traffic is shown.
> Can I perform any other test about this?
>

First, have you tried to leave vtnet1 alone and try:

# ifconfig vtnet0 up 192.168.1.x -txcsum -rxcsum -tso4 -tso6
# vale-ctl -h vale0:vtnet0
and verified that you can ssh into 192.168.1.x from the host?

At least you verify that the vale-ctl -h works also on your enivironment.

Cheers,
  Vincenzo


> Cheers,
> Antho
>
> Il giorno mer 3 giu 2020 alle ore 19:53 Vincenzo Maffione <
> vmaffi...@freebsd.org> ha scritto:
>
>> Hi Anthony,
>>   I fixed more bugs in the vtnet driver (in FreeBSD-CURRENT). As of
>> r361760, I'm now able to run the following steps in a VM with a vtnet device
>>
>> # ifconfig vtnet0 -txcsum -rxcsum -tso4 192.168.100.2/24
>> # vale-ctl -h vale:vtnet0
>> # netperf -H 192.168.100.1  # Run a netperf TCP_STREAM test to the host
>> bridge interface (br0)
>>
>> Since TCP works correctly at reasonable speed I'm confident that most of
>> the existing problems have gone.
>> Let me know if you have any questions about this.
>>
>> Cheers,
>>   Vincenzo
>>
>> Il giorno lun 1 giu 2020 alle ore 18:22 Vincenzo Maffione <
>> vmaffi...@freebsd.org> ha scritto:
>>
>>> Hi Anthony,
>>>   I think there is more than a bug, drivers-related, that show up when
>>> you attach the interface to a vale switch.
>>> I've found and fixed some related to if_vtnet (see below). In any case,
>>> in my tests there is no difference between TCP traffic and other (UDP,
>>> ICMP, STP,...).
>>> The issues are not related to LRO, as I thought.
>>> There are still more bugs in vtnet and I'm trying to chase them.
>>> In the meanwhile it would help if you apply the patches below and try
>>> again with vtnet to see if the situation improves. They apply cleanly to
>>> 12.1 release.
>>>
>>> Regarding your problem with em devices, it is probably yet a different
>>> issue. It may be related to the iflib transition or not. It would help to
>>> try the same setup on stable/11 (which does not have iflib). I don't have
>>> an em device, but I will try with an emulated em in QEMU/KVM.
>>>
>>> Cheers,
>>>   Vincenzo
>>>
>>> 
>>> r361698 | vmaffione | 2020-06-01 16:14:29 + (Mon, 01 Jun 2020) | 8
>>> lines
>>>
>>> netmap: if_vtnet: avoid netmap ring wraparound
>>>
>>> netmap assumes the one "slot" is left unused to distinguish
>>> the empty ring and full ring conditions. This assumption was
>>> violated by vtnet_netmap_rxq_populate().
>>>
>>> MFC after:  1 week
>>>
>>> 
>>> r361697 | vmaffione | 2020-06-01 16:12:09 + (Mon, 01 Jun 2020) | 8
>>> lines
>>>
>>> netmap: if_vtnet: replace vtnet_free_used()
>>>
>>> The functionality contained in this function is duplicated,
>>> as it is already available in vtnet_txq_free_mbufs()
>>> and vtnet_rxq_free_mbufs().
>>>
>>> MFC after:  1 week
>>>
>>> 
>>> r361696 | vmaffione | 2020-06-01 16:10:44 + (Mon, 01 Jun 2020) | 13
>>> lines
>>>
>>> netmap: vtnet: fix RX virtqueue initialization bug
>>>
>>> The vtnet_netmap_rxq_populate() function erroneously assumed
>>> that kring->nr_hwcur = 0, i.e. the kring was in the initial
>>> state. However, this is not always the case: for example,
>>> when a vtnet reinit is triggered by some changes in the
>>> interface flags or capenable.
>>> This patch changes the behaviour of vtnet_netmap_kring_refill()
>>> so that it always starts publishing the netmap buffers starting
>>> from the current value of kring->nr_hwcur.
>>>
>>> MFC after:  1 week
>>> 
>>> Il giorno lun 1 giu 2020 alle ore 15:19 Anthony Arnaud <
>>> antho.arnaudi...@gmail.com> ha scritto:
>>>
 Hi Vincenzo,

 To simplify the scenario I have installed from scratch FBSD12.1 on a
 new machine, without any virtualization env.
 I have encountered the same problem, when i attach an ethernet
 interface to vale switch (in this case an intel card em5) the tcp traffic
 disappears and tcpdump shown only udp, icmp6 and stp packets.
 If 

Re: Netmap - Vale switch - tcp problem

[Anthony Arnaud]

Hi Vincenzo,
Thank you for your answers.
I upgraded a guest machine in proxmox env to FreeBSD 13.0-CURRENT #0
r361830.
After that I compiled the tools as usual from /src/tools/tools/netmap
I configured 2 NIC vtnet

vtnet0 with an ip 192.168.1.x
vtnet1 without ip and csum disabled as mirror of vtnet0 (using Open vSwitch)
Tcp traffic is generated by an ssh connection from my host to guest
machine.

"tcpdump -i vtnet1 tcp": at each keypress in ssh shell connected to
192.168.1.x I see few tcp packet sniffed in guest machine.
But if I attach vtnet1 to vale switch, tcpdump no longer works as before.
No tcp traffic is shown.
Can I perform any other test about this?

Cheers,
Antho

Il giorno mer 3 giu 2020 alle ore 19:53 Vincenzo Maffione <
vmaffi...@freebsd.org> ha scritto:

> Hi Anthony,
>   I fixed more bugs in the vtnet driver (in FreeBSD-CURRENT). As of
> r361760, I'm now able to run the following steps in a VM with a vtnet device
>
> # ifconfig vtnet0 -txcsum -rxcsum -tso4 192.168.100.2/24
> # vale-ctl -h vale:vtnet0
> # netperf -H 192.168.100.1  # Run a netperf TCP_STREAM test to the host
> bridge interface (br0)
>
> Since TCP works correctly at reasonable speed I'm confident that most of
> the existing problems have gone.
> Let me know if you have any questions about this.
>
> Cheers,
>   Vincenzo
>
> Il giorno lun 1 giu 2020 alle ore 18:22 Vincenzo Maffione <
> vmaffi...@freebsd.org> ha scritto:
>
>> Hi Anthony,
>>   I think there is more than a bug, drivers-related, that show up when
>> you attach the interface to a vale switch.
>> I've found and fixed some related to if_vtnet (see below). In any case,
>> in my tests there is no difference between TCP traffic and other (UDP,
>> ICMP, STP,...).
>> The issues are not related to LRO, as I thought.
>> There are still more bugs in vtnet and I'm trying to chase them.
>> In the meanwhile it would help if you apply the patches below and try
>> again with vtnet to see if the situation improves. They apply cleanly to
>> 12.1 release.
>>
>> Regarding your problem with em devices, it is probably yet a different
>> issue. It may be related to the iflib transition or not. It would help to
>> try the same setup on stable/11 (which does not have iflib). I don't have
>> an em device, but I will try with an emulated em in QEMU/KVM.
>>
>> Cheers,
>>   Vincenzo
>>
>> 
>> r361698 | vmaffione | 2020-06-01 16:14:29 + (Mon, 01 Jun 2020) | 8
>> lines
>>
>> netmap: if_vtnet: avoid netmap ring wraparound
>>
>> netmap assumes the one "slot" is left unused to distinguish
>> the empty ring and full ring conditions. This assumption was
>> violated by vtnet_netmap_rxq_populate().
>>
>> MFC after:  1 week
>>
>> 
>> r361697 | vmaffione | 2020-06-01 16:12:09 + (Mon, 01 Jun 2020) | 8
>> lines
>>
>> netmap: if_vtnet: replace vtnet_free_used()
>>
>> The functionality contained in this function is duplicated,
>> as it is already available in vtnet_txq_free_mbufs()
>> and vtnet_rxq_free_mbufs().
>>
>> MFC after:  1 week
>>
>> 
>> r361696 | vmaffione | 2020-06-01 16:10:44 + (Mon, 01 Jun 2020) | 13
>> lines
>>
>> netmap: vtnet: fix RX virtqueue initialization bug
>>
>> The vtnet_netmap_rxq_populate() function erroneously assumed
>> that kring->nr_hwcur = 0, i.e. the kring was in the initial
>> state. However, this is not always the case: for example,
>> when a vtnet reinit is triggered by some changes in the
>> interface flags or capenable.
>> This patch changes the behaviour of vtnet_netmap_kring_refill()
>> so that it always starts publishing the netmap buffers starting
>> from the current value of kring->nr_hwcur.
>>
>> MFC after:  1 week
>> 
>> Il giorno lun 1 giu 2020 alle ore 15:19 Anthony Arnaud <
>> antho.arnaudi...@gmail.com> ha scritto:
>>
>>> Hi Vincenzo,
>>>
>>> To simplify the scenario I have installed from scratch FBSD12.1 on a new
>>> machine, without any virtualization env.
>>> I have encountered the same problem, when i attach an ethernet interface
>>> to vale switch (in this case an intel card em5) the tcp traffic disappears
>>> and tcpdump shown only udp, icmp6 and stp packets.
>>> If I detach the NIC from vale0 tcpdump shown all tcp traffic.
>>> I'm using the netmap version included in FBSD 12.1, and I have compiled
>>> vale-ctl presents in kernel sources (/src/tools/tools/netmap/)
>>> I executed your steps.
>>> There is something dark about that behaviour...
>>>
>>> Cheers
>>> Anthon
>>>
>>>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap - Vale switch - tcp problem

[Vincenzo Maffione]

Hi Anthony,
  I fixed more bugs in the vtnet driver (in FreeBSD-CURRENT). As of
r361760, I'm now able to run the following steps in a VM with a vtnet device

# ifconfig vtnet0 -txcsum -rxcsum -tso4 192.168.100.2/24
# vale-ctl -h vale:vtnet0
# netperf -H 192.168.100.1  # Run a netperf TCP_STREAM test to the host
bridge interface (br0)

Since TCP works correctly at reasonable speed I'm confident that most of
the existing problems have gone.
Let me know if you have any questions about this.

Cheers,
  Vincenzo

Il giorno lun 1 giu 2020 alle ore 18:22 Vincenzo Maffione <
vmaffi...@freebsd.org> ha scritto:

> Hi Anthony,
>   I think there is more than a bug, drivers-related, that show up when you
> attach the interface to a vale switch.
> I've found and fixed some related to if_vtnet (see below). In any case, in
> my tests there is no difference between TCP traffic and other (UDP, ICMP,
> STP,...).
> The issues are not related to LRO, as I thought.
> There are still more bugs in vtnet and I'm trying to chase them.
> In the meanwhile it would help if you apply the patches below and try
> again with vtnet to see if the situation improves. They apply cleanly to
> 12.1 release.
>
> Regarding your problem with em devices, it is probably yet a different
> issue. It may be related to the iflib transition or not. It would help to
> try the same setup on stable/11 (which does not have iflib). I don't have
> an em device, but I will try with an emulated em in QEMU/KVM.
>
> Cheers,
>   Vincenzo
>
> 
> r361698 | vmaffione | 2020-06-01 16:14:29 + (Mon, 01 Jun 2020) | 8
> lines
>
> netmap: if_vtnet: avoid netmap ring wraparound
>
> netmap assumes the one "slot" is left unused to distinguish
> the empty ring and full ring conditions. This assumption was
> violated by vtnet_netmap_rxq_populate().
>
> MFC after:  1 week
>
> 
> r361697 | vmaffione | 2020-06-01 16:12:09 + (Mon, 01 Jun 2020) | 8
> lines
>
> netmap: if_vtnet: replace vtnet_free_used()
>
> The functionality contained in this function is duplicated,
> as it is already available in vtnet_txq_free_mbufs()
> and vtnet_rxq_free_mbufs().
>
> MFC after:  1 week
>
> 
> r361696 | vmaffione | 2020-06-01 16:10:44 + (Mon, 01 Jun 2020) | 13
> lines
>
> netmap: vtnet: fix RX virtqueue initialization bug
>
> The vtnet_netmap_rxq_populate() function erroneously assumed
> that kring->nr_hwcur = 0, i.e. the kring was in the initial
> state. However, this is not always the case: for example,
> when a vtnet reinit is triggered by some changes in the
> interface flags or capenable.
> This patch changes the behaviour of vtnet_netmap_kring_refill()
> so that it always starts publishing the netmap buffers starting
> from the current value of kring->nr_hwcur.
>
> MFC after:  1 week
> 
> Il giorno lun 1 giu 2020 alle ore 15:19 Anthony Arnaud <
> antho.arnaudi...@gmail.com> ha scritto:
>
>> Hi Vincenzo,
>>
>> To simplify the scenario I have installed from scratch FBSD12.1 on a new
>> machine, without any virtualization env.
>> I have encountered the same problem, when i attach an ethernet interface
>> to vale switch (in this case an intel card em5) the tcp traffic disappears
>> and tcpdump shown only udp, icmp6 and stp packets.
>> If I detach the NIC from vale0 tcpdump shown all tcp traffic.
>> I'm using the netmap version included in FBSD 12.1, and I have compiled
>> vale-ctl presents in kernel sources (/src/tools/tools/netmap/)
>> I executed your steps.
>> There is something dark about that behaviour...
>>
>> Cheers
>> Anthon
>>
>>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap - Vale switch - tcp problem

[Vincenzo Maffione]

Hi Anthony,
  I think there is more than a bug, drivers-related, that show up when you
attach the interface to a vale switch.
I've found and fixed some related to if_vtnet (see below). In any case, in
my tests there is no difference between TCP traffic and other (UDP, ICMP,
STP,...).
The issues are not related to LRO, as I thought.
There are still more bugs in vtnet and I'm trying to chase them.
In the meanwhile it would help if you apply the patches below and try again
with vtnet to see if the situation improves. They apply cleanly to 12.1
release.

Regarding your problem with em devices, it is probably yet a different
issue. It may be related to the iflib transition or not. It would help to
try the same setup on stable/11 (which does not have iflib). I don't have
an em device, but I will try with an emulated em in QEMU/KVM.

Cheers,
  Vincenzo


r361698 | vmaffione | 2020-06-01 16:14:29 + (Mon, 01 Jun 2020) | 8 lines

netmap: if_vtnet: avoid netmap ring wraparound

netmap assumes the one "slot" is left unused to distinguish
the empty ring and full ring conditions. This assumption was
violated by vtnet_netmap_rxq_populate().

MFC after:  1 week


r361697 | vmaffione | 2020-06-01 16:12:09 + (Mon, 01 Jun 2020) | 8 lines

netmap: if_vtnet: replace vtnet_free_used()

The functionality contained in this function is duplicated,
as it is already available in vtnet_txq_free_mbufs()
and vtnet_rxq_free_mbufs().

MFC after:  1 week


r361696 | vmaffione | 2020-06-01 16:10:44 + (Mon, 01 Jun 2020) | 13
lines

netmap: vtnet: fix RX virtqueue initialization bug

The vtnet_netmap_rxq_populate() function erroneously assumed
that kring->nr_hwcur = 0, i.e. the kring was in the initial
state. However, this is not always the case: for example,
when a vtnet reinit is triggered by some changes in the
interface flags or capenable.
This patch changes the behaviour of vtnet_netmap_kring_refill()
so that it always starts publishing the netmap buffers starting
from the current value of kring->nr_hwcur.

MFC after:  1 week

Il giorno lun 1 giu 2020 alle ore 15:19 Anthony Arnaud <
antho.arnaudi...@gmail.com> ha scritto:

> Hi Vincenzo,
>
> To simplify the scenario I have installed from scratch FBSD12.1 on a new
> machine, without any virtualization env.
> I have encountered the same problem, when i attach an ethernet interface
> to vale switch (in this case an intel card em5) the tcp traffic disappears
> and tcpdump shown only udp, icmp6 and stp packets.
> If I detach the NIC from vale0 tcpdump shown all tcp traffic.
> I'm using the netmap version included in FBSD 12.1, and I have compiled
> vale-ctl presents in kernel sources (/src/tools/tools/netmap/)
> I executed your steps.
> There is something dark about that behaviour...
>
> Cheers
> Anthon
>
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap - Vale switch - tcp problem

[Anthony Arnaud]

Hi Vincenzo,

To simplify the scenario I have installed from scratch FBSD12.1 on a new
machine, without any virtualization env.
I have encountered the same problem, when i attach an ethernet interface to
vale switch (in this case an intel card em5) the tcp traffic disappears and
tcpdump shown only udp, icmp6 and stp packets.
If I detach the NIC from vale0 tcpdump shown all tcp traffic.
I'm using the netmap version included in FBSD 12.1, and I have compiled
vale-ctl presents in kernel sources (/src/tools/tools/netmap/)
I executed your steps.
There is something dark about that behaviour...

Cheers
Anthon

Il giorno sab 30 mag 2020 alle ore 14:29 Vincenzo Maffione <
vmaffi...@freebsd.org> ha scritto:

>
>
> Il giorno ven 29 mag 2020 alle ore 19:01 Anthony Arnaud <
> antho.arnaudi...@gmail.com> ha scritto:
>
>> Hi Vincenzo,
>>
>> thanks for your hints!
>> I rebooted my guest FBSD 12.1 machine, and I have perfomed your steps
>>
>> #ifconfig vtnet1 -txcsum -rxcsum -tso4 -tso6 up promisc
>>
>> vtnet1:
>> flags=28943 metric
>> 0 mtu 1500
>>
>> options=6c04b8
>> ether 0e:bd:ec:7a:08:06
>> media: Ethernet 10Gbase-T 
>> status: active
>> nd6 options=29
>>
>> tcpdump is ok.
>>
>> Does it mean you see both ICMP, UDP and TCP traffic?
>
>
>> But after:
>>
>> #vale-ctl
>>
>> 446.196827 bdg_ctl [149] bridge:0 port:0 vale0:vtnet1
>> 446.196855 bdg_ctl [149] bridge:0 port:1 vale0:vtnet1^
>>
>> tcpdump not work anymore.
>>
>
> Do you see ICMP/UDP only and not TCP?
>
>
>> I don't understand why the tcp traffic disappears.
>> In my configuration vtnet1 is a mirror port created by Open vSwitch, but
>> I don't think that's the reason.
>>
>> No, I don't think that's relevant.
>
> In my setup, vtnet0 is a guest interface backed by a host tap device
> (attached to a linux bridge), and hypervisor is QEMU/KVM.
> Here are the steps I follow in the VM (in this order):
>
> # ifconfig vtnet0 -txcsum -rxcsum -tso4 -tso6 up 192.168.100.2/24
> # vale-ctl -h vale0:vtnet0
>
> # nc 192.168.100.1  # connect to listening netcat in the host.
> hello
> abc
> [...]
> # tcpdump -ni vtnet0 # This shows the TCP traffic.
>
> I start to see problems when I change the offloads:
> # ifconfig vtnet0 -lro
>
> Cheers,
>   Vincenzo
>
>
>> Below some info about my configuration:
>>
>> dev.netmap.iflib_rx_miss_bufs: 0
>> dev.netmap.iflib_rx_miss: 0
>> dev.netmap.iflib_crcstrip: 1
>> dev.netmap.bridge_batch: 1024
>> dev.netmap.default_pipes: 0
>> dev.netmap.priv_buf_num: 4098
>> dev.netmap.priv_buf_size: 2048
>> dev.netmap.buf_curr_num: 163840
>> dev.netmap.buf_num: 163840
>> dev.netmap.buf_curr_size: 2048
>> dev.netmap.buf_size: 2048
>> dev.netmap.priv_ring_num: 4
>> dev.netmap.priv_ring_size: 20480
>> dev.netmap.ring_curr_num: 200
>> dev.netmap.ring_num: 200
>> dev.netmap.ring_curr_size: 36864
>> dev.netmap.ring_size: 36864
>> dev.netmap.priv_if_num: 2
>> dev.netmap.priv_if_size: 1024
>> dev.netmap.if_curr_num: 100
>> dev.netmap.if_num: 100
>> dev.netmap.if_curr_size: 1024
>> dev.netmap.if_size: 1024
>> dev.netmap.ptnet_vnet_hdr: 1
>> dev.netmap.generic_rings: 1
>> dev.netmap.generic_ringsize: 1024
>> dev.netmap.generic_mit: 10
>> dev.netmap.generic_hwcsum: 0
>> dev.netmap.admode: 0
>> dev.netmap.fwd: 0
>> dev.netmap.txsync_retry: 2
>> dev.netmap.no_pendintr: 1
>> dev.netmap.no_timestamp: 0
>> dev.netmap.verbose: 0
>>
>>
>> dev.vtnet.1.txq0.rescheduled: 0
>> dev.vtnet.1.txq0.tso: 0
>> dev.vtnet.1.txq0.csum: 0
>> dev.vtnet.1.txq0.omcasts: 0
>> dev.vtnet.1.txq0.obytes: 0
>> dev.vtnet.1.txq0.opackets: 0
>> dev.vtnet.1.rxq0.rescheduled: 0
>> dev.vtnet.1.rxq0.csum_failed: 0
>> dev.vtnet.1.rxq0.csum: 66
>> dev.vtnet.1.rxq0.ierrors: 0
>> dev.vtnet.1.rxq0.iqdrops: 0
>> dev.vtnet.1.rxq0.ibytes: 11904780
>> dev.vtnet.1.rxq0.ipackets: 40984
>> dev.vtnet.1.tx_task_rescheduled: 0
>> dev.vtnet.1.tx_tso_offloaded: 0
>> dev.vtnet.1.tx_csum_offloaded: 0
>> dev.vtnet.1.tx_defrag_failed: 0
>> dev.vtnet.1.tx_defragged: 0
>> dev.vtnet.1.tx_tso_not_tcp: 0
>> dev.vtnet.1.tx_tso_bad_ethtype: 0
>> dev.vtnet.1.tx_csum_bad_ethtype: 0
>> dev.vtnet.1.rx_task_rescheduled: 0
>> dev.vtnet.1.rx_csum_offloaded: 0
>> dev.vtnet.1.rx_csum_failed: 0
>> dev.vtnet.1.rx_csum_bad_proto: 0
>> dev.vtnet.1.rx_csum_bad_offset: 0
>> dev.vtnet.1.rx_csum_bad_ipproto: 0
>> dev.vtnet.1.rx_csum_bad_ethtype: 0
>> dev.vtnet.1.rx_mergeable_failed: 0
>> dev.vtnet.1.rx_enq_replacement_failed: 0
>> dev.vtnet.1.rx_frame_too_large: 0
>> dev.vtnet.1.mbuf_alloc_failed: 0
>> dev.vtnet.1.act_vq_pairs: 1
>> dev.vtnet.1.requested_vq_pairs: 0
>> dev.vtnet.1.max_vq_pairs: 1
>> dev.vtnet.1.%parent: virtio_pci4
>> dev.vtnet.1.%pnpinfo:
>> dev.vtnet.1.%location:
>> dev.vtnet.1.%driver: vtnet
>> dev.vtnet.1.%desc: VirtIO Networking Adapter
>> dev.vtnet.0.txq0.rescheduled: 0
>>
>> Cheers,
>> Anthony
>>
>> Il giorno gio 28 mag 2020 alle ore 21:38 Vincenzo Maffione <
>> vmaffi...@freebsd.org> ha scritto:
>>
>>> Hi,
>>>   I was trying to reproduce your problem (same FreeBSD release as
>>> 

Re: Netmap - Vale switch - tcp problem

[Vincenzo Maffione]

Il giorno ven 29 mag 2020 alle ore 19:01 Anthony Arnaud <
antho.arnaudi...@gmail.com> ha scritto:

> Hi Vincenzo,
>
> thanks for your hints!
> I rebooted my guest FBSD 12.1 machine, and I have perfomed your steps
>
> #ifconfig vtnet1 -txcsum -rxcsum -tso4 -tso6 up promisc
>
> vtnet1:
> flags=28943 metric
> 0 mtu 1500
>
> options=6c04b8
> ether 0e:bd:ec:7a:08:06
> media: Ethernet 10Gbase-T 
> status: active
> nd6 options=29
>
> tcpdump is ok.
>
> Does it mean you see both ICMP, UDP and TCP traffic?


> But after:
>
> #vale-ctl
>
> 446.196827 bdg_ctl [149] bridge:0 port:0 vale0:vtnet1
> 446.196855 bdg_ctl [149] bridge:0 port:1 vale0:vtnet1^
>
> tcpdump not work anymore.
>

Do you see ICMP/UDP only and not TCP?


> I don't understand why the tcp traffic disappears.
> In my configuration vtnet1 is a mirror port created by Open vSwitch, but I
> don't think that's the reason.
>
> No, I don't think that's relevant.

In my setup, vtnet0 is a guest interface backed by a host tap device
(attached to a linux bridge), and hypervisor is QEMU/KVM.
Here are the steps I follow in the VM (in this order):

# ifconfig vtnet0 -txcsum -rxcsum -tso4 -tso6 up 192.168.100.2/24
# vale-ctl -h vale0:vtnet0

# nc 192.168.100.1  # connect to listening netcat in the host.
hello
abc
[...]
# tcpdump -ni vtnet0 # This shows the TCP traffic.

I start to see problems when I change the offloads:
# ifconfig vtnet0 -lro

Cheers,
  Vincenzo


> Below some info about my configuration:
>
> dev.netmap.iflib_rx_miss_bufs: 0
> dev.netmap.iflib_rx_miss: 0
> dev.netmap.iflib_crcstrip: 1
> dev.netmap.bridge_batch: 1024
> dev.netmap.default_pipes: 0
> dev.netmap.priv_buf_num: 4098
> dev.netmap.priv_buf_size: 2048
> dev.netmap.buf_curr_num: 163840
> dev.netmap.buf_num: 163840
> dev.netmap.buf_curr_size: 2048
> dev.netmap.buf_size: 2048
> dev.netmap.priv_ring_num: 4
> dev.netmap.priv_ring_size: 20480
> dev.netmap.ring_curr_num: 200
> dev.netmap.ring_num: 200
> dev.netmap.ring_curr_size: 36864
> dev.netmap.ring_size: 36864
> dev.netmap.priv_if_num: 2
> dev.netmap.priv_if_size: 1024
> dev.netmap.if_curr_num: 100
> dev.netmap.if_num: 100
> dev.netmap.if_curr_size: 1024
> dev.netmap.if_size: 1024
> dev.netmap.ptnet_vnet_hdr: 1
> dev.netmap.generic_rings: 1
> dev.netmap.generic_ringsize: 1024
> dev.netmap.generic_mit: 10
> dev.netmap.generic_hwcsum: 0
> dev.netmap.admode: 0
> dev.netmap.fwd: 0
> dev.netmap.txsync_retry: 2
> dev.netmap.no_pendintr: 1
> dev.netmap.no_timestamp: 0
> dev.netmap.verbose: 0
>
>
> dev.vtnet.1.txq0.rescheduled: 0
> dev.vtnet.1.txq0.tso: 0
> dev.vtnet.1.txq0.csum: 0
> dev.vtnet.1.txq0.omcasts: 0
> dev.vtnet.1.txq0.obytes: 0
> dev.vtnet.1.txq0.opackets: 0
> dev.vtnet.1.rxq0.rescheduled: 0
> dev.vtnet.1.rxq0.csum_failed: 0
> dev.vtnet.1.rxq0.csum: 66
> dev.vtnet.1.rxq0.ierrors: 0
> dev.vtnet.1.rxq0.iqdrops: 0
> dev.vtnet.1.rxq0.ibytes: 11904780
> dev.vtnet.1.rxq0.ipackets: 40984
> dev.vtnet.1.tx_task_rescheduled: 0
> dev.vtnet.1.tx_tso_offloaded: 0
> dev.vtnet.1.tx_csum_offloaded: 0
> dev.vtnet.1.tx_defrag_failed: 0
> dev.vtnet.1.tx_defragged: 0
> dev.vtnet.1.tx_tso_not_tcp: 0
> dev.vtnet.1.tx_tso_bad_ethtype: 0
> dev.vtnet.1.tx_csum_bad_ethtype: 0
> dev.vtnet.1.rx_task_rescheduled: 0
> dev.vtnet.1.rx_csum_offloaded: 0
> dev.vtnet.1.rx_csum_failed: 0
> dev.vtnet.1.rx_csum_bad_proto: 0
> dev.vtnet.1.rx_csum_bad_offset: 0
> dev.vtnet.1.rx_csum_bad_ipproto: 0
> dev.vtnet.1.rx_csum_bad_ethtype: 0
> dev.vtnet.1.rx_mergeable_failed: 0
> dev.vtnet.1.rx_enq_replacement_failed: 0
> dev.vtnet.1.rx_frame_too_large: 0
> dev.vtnet.1.mbuf_alloc_failed: 0
> dev.vtnet.1.act_vq_pairs: 1
> dev.vtnet.1.requested_vq_pairs: 0
> dev.vtnet.1.max_vq_pairs: 1
> dev.vtnet.1.%parent: virtio_pci4
> dev.vtnet.1.%pnpinfo:
> dev.vtnet.1.%location:
> dev.vtnet.1.%driver: vtnet
> dev.vtnet.1.%desc: VirtIO Networking Adapter
> dev.vtnet.0.txq0.rescheduled: 0
>
> Cheers,
> Anthony
>
> Il giorno gio 28 mag 2020 alle ore 21:38 Vincenzo Maffione <
> vmaffi...@freebsd.org> ha scritto:
>
>> Hi,
>>   I was trying to reproduce your problem (same FreeBSD release as yours).
>> It looks like there is some sort of bad interaction with LRO.
>>
>> Starting from a fresh boot, if you keep lro enabled, e.g.
>>   # ifconfig vtnet0 -txcsum -rxcsum -tso4 -tso6
>>   # vale-ctl 
>> then I experience no problem (TCP works between vtnet0 and the host,
>> tcpdump on vtnet0 works as expected).
>>
>> As soon as you disable LRO:
>>   # ifconfig vtnet0 -lro
>> both TCP and tcpdump stop working.
>> If I enable LRO again, TCP restarts working, but tcpdump doesn't. I need
>> to reboot the machine to fix it.
>>
>> Btw, creating vi0 (persistent VALE port) is not relevant for this test.
>> You may as well use ephemeral VALE ports (e.g. run pkt-gen -i vale0:1 -f
>> rx).
>>
>> I will have a look at the LRO issue asap. In the meantime you could avoid
>> disabling LRO and see if that works for you.
>>
>> Cheers,
>>   Vincenzo
>>
>> Il giorno gio 28 mag 2020 alle 

Re: Netmap - Vale switch - tcp problem

[Anthony Arnaud]

Hi Vincenzo,

thanks for your hints!
I rebooted my guest FBSD 12.1 machine, and I have perfomed your steps

#ifconfig vtnet1 -txcsum -rxcsum -tso4 -tso6 up promisc

vtnet1:
flags=28943 metric
0 mtu 1500

options=6c04b8
ether 0e:bd:ec:7a:08:06
media: Ethernet 10Gbase-T 
status: active
nd6 options=29

tcpdump is ok.

But after:

#vale-ctl

446.196827 bdg_ctl [149] bridge:0 port:0 vale0:vtnet1
446.196855 bdg_ctl [149] bridge:0 port:1 vale0:vtnet1^

tcpdump not work anymore.

I don't understand why the tcp traffic disappears.
In my configuration vtnet1 is a mirror port created by Open vSwitch, but I
don't think that's the reason.

Below some info about my configuration:

dev.netmap.iflib_rx_miss_bufs: 0
dev.netmap.iflib_rx_miss: 0
dev.netmap.iflib_crcstrip: 1
dev.netmap.bridge_batch: 1024
dev.netmap.default_pipes: 0
dev.netmap.priv_buf_num: 4098
dev.netmap.priv_buf_size: 2048
dev.netmap.buf_curr_num: 163840
dev.netmap.buf_num: 163840
dev.netmap.buf_curr_size: 2048
dev.netmap.buf_size: 2048
dev.netmap.priv_ring_num: 4
dev.netmap.priv_ring_size: 20480
dev.netmap.ring_curr_num: 200
dev.netmap.ring_num: 200
dev.netmap.ring_curr_size: 36864
dev.netmap.ring_size: 36864
dev.netmap.priv_if_num: 2
dev.netmap.priv_if_size: 1024
dev.netmap.if_curr_num: 100
dev.netmap.if_num: 100
dev.netmap.if_curr_size: 1024
dev.netmap.if_size: 1024
dev.netmap.ptnet_vnet_hdr: 1
dev.netmap.generic_rings: 1
dev.netmap.generic_ringsize: 1024
dev.netmap.generic_mit: 10
dev.netmap.generic_hwcsum: 0
dev.netmap.admode: 0
dev.netmap.fwd: 0
dev.netmap.txsync_retry: 2
dev.netmap.no_pendintr: 1
dev.netmap.no_timestamp: 0
dev.netmap.verbose: 0


dev.vtnet.1.txq0.rescheduled: 0
dev.vtnet.1.txq0.tso: 0
dev.vtnet.1.txq0.csum: 0
dev.vtnet.1.txq0.omcasts: 0
dev.vtnet.1.txq0.obytes: 0
dev.vtnet.1.txq0.opackets: 0
dev.vtnet.1.rxq0.rescheduled: 0
dev.vtnet.1.rxq0.csum_failed: 0
dev.vtnet.1.rxq0.csum: 66
dev.vtnet.1.rxq0.ierrors: 0
dev.vtnet.1.rxq0.iqdrops: 0
dev.vtnet.1.rxq0.ibytes: 11904780
dev.vtnet.1.rxq0.ipackets: 40984
dev.vtnet.1.tx_task_rescheduled: 0
dev.vtnet.1.tx_tso_offloaded: 0
dev.vtnet.1.tx_csum_offloaded: 0
dev.vtnet.1.tx_defrag_failed: 0
dev.vtnet.1.tx_defragged: 0
dev.vtnet.1.tx_tso_not_tcp: 0
dev.vtnet.1.tx_tso_bad_ethtype: 0
dev.vtnet.1.tx_csum_bad_ethtype: 0
dev.vtnet.1.rx_task_rescheduled: 0
dev.vtnet.1.rx_csum_offloaded: 0
dev.vtnet.1.rx_csum_failed: 0
dev.vtnet.1.rx_csum_bad_proto: 0
dev.vtnet.1.rx_csum_bad_offset: 0
dev.vtnet.1.rx_csum_bad_ipproto: 0
dev.vtnet.1.rx_csum_bad_ethtype: 0
dev.vtnet.1.rx_mergeable_failed: 0
dev.vtnet.1.rx_enq_replacement_failed: 0
dev.vtnet.1.rx_frame_too_large: 0
dev.vtnet.1.mbuf_alloc_failed: 0
dev.vtnet.1.act_vq_pairs: 1
dev.vtnet.1.requested_vq_pairs: 0
dev.vtnet.1.max_vq_pairs: 1
dev.vtnet.1.%parent: virtio_pci4
dev.vtnet.1.%pnpinfo:
dev.vtnet.1.%location:
dev.vtnet.1.%driver: vtnet
dev.vtnet.1.%desc: VirtIO Networking Adapter
dev.vtnet.0.txq0.rescheduled: 0

Cheers,
Anthony

Il giorno gio 28 mag 2020 alle ore 21:38 Vincenzo Maffione <
vmaffi...@freebsd.org> ha scritto:

> Hi,
>   I was trying to reproduce your problem (same FreeBSD release as yours).
> It looks like there is some sort of bad interaction with LRO.
>
> Starting from a fresh boot, if you keep lro enabled, e.g.
>   # ifconfig vtnet0 -txcsum -rxcsum -tso4 -tso6
>   # vale-ctl 
> then I experience no problem (TCP works between vtnet0 and the host,
> tcpdump on vtnet0 works as expected).
>
> As soon as you disable LRO:
>   # ifconfig vtnet0 -lro
> both TCP and tcpdump stop working.
> If I enable LRO again, TCP restarts working, but tcpdump doesn't. I need
> to reboot the machine to fix it.
>
> Btw, creating vi0 (persistent VALE port) is not relevant for this test.
> You may as well use ephemeral VALE ports (e.g. run pkt-gen -i vale0:1 -f
> rx).
>
> I will have a look at the LRO issue asap. In the meantime you could avoid
> disabling LRO and see if that works for you.
>
> Cheers,
>   Vincenzo
>
> Il giorno gio 28 mag 2020 alle ore 17:16 Anthony Arnaud <
> antho.arnaudi...@gmail.com> ha scritto:
>
>> I already disabled the checksum, the vtnet config is:
>>
>> ifconfig vtnet1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6
>> -vlanmtu
>> -vlanhwtag -vlanhwfilter -vlanhwtso -vlanhwcsum up promisc
>>
>> vtnet1:
>> flags=28943
>> metric
>> 0 mtu 1500
>> options=1800a8
>> ether 0e:bd:ec:7a:08:06
>> media: Ethernet 10Gbase-T 
>> status: active
>> nd6 options=29
>>
>> Sorry for not having posted vtnet config before.
>> PS: VLAN_HWCSUM is not switchable off for some reason!
>> the problem is not that.
>>
>> Cheers
>> Anthony
>>
>>
>>
>>
>> Il giorno gio 28 mag 2020 alle ore 16:05 Luigi Rizzo 
>> ha scritto:
>>
>> >
>> >
>> > On Thursday, May 28, 2020, Anthony Arnaud 
>> > wrote:
>> >
>> >> Hi everyone!
>> >> I would like to create a vale switch with an interface attached with
>> the
>> >> host stack and some virtual.
>> >> My env is a VM with FBSD-12.1 12.1-RELEASE 

Re: Netmap - Vale switch - tcp problem

[Vincenzo Maffione]

Hi,
  I was trying to reproduce your problem (same FreeBSD release as yours).
It looks like there is some sort of bad interaction with LRO.

Starting from a fresh boot, if you keep lro enabled, e.g.
  # ifconfig vtnet0 -txcsum -rxcsum -tso4 -tso6
  # vale-ctl 
then I experience no problem (TCP works between vtnet0 and the host,
tcpdump on vtnet0 works as expected).

As soon as you disable LRO:
  # ifconfig vtnet0 -lro
both TCP and tcpdump stop working.
If I enable LRO again, TCP restarts working, but tcpdump doesn't. I need to
reboot the machine to fix it.

Btw, creating vi0 (persistent VALE port) is not relevant for this test. You
may as well use ephemeral VALE ports (e.g. run pkt-gen -i vale0:1 -f rx).

I will have a look at the LRO issue asap. In the meantime you could avoid
disabling LRO and see if that works for you.

Cheers,
  Vincenzo

Il giorno gio 28 mag 2020 alle ore 17:16 Anthony Arnaud <
antho.arnaudi...@gmail.com> ha scritto:

> I already disabled the checksum, the vtnet config is:
>
> ifconfig vtnet1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6 -vlanmtu
> -vlanhwtag -vlanhwfilter -vlanhwtso -vlanhwcsum up promisc
>
> vtnet1:
> flags=28943 metric
> 0 mtu 1500
> options=1800a8
> ether 0e:bd:ec:7a:08:06
> media: Ethernet 10Gbase-T 
> status: active
> nd6 options=29
>
> Sorry for not having posted vtnet config before.
> PS: VLAN_HWCSUM is not switchable off for some reason!
> the problem is not that.
>
> Cheers
> Anthony
>
>
>
>
> Il giorno gio 28 mag 2020 alle ore 16:05 Luigi Rizzo 
> ha scritto:
>
> >
> >
> > On Thursday, May 28, 2020, Anthony Arnaud 
> > wrote:
> >
> >> Hi everyone!
> >> I would like to create a vale switch with an interface attached with the
> >> host stack and some virtual.
> >> My env is a VM with FBSD-12.1 12.1-RELEASE FreeBSD 12.1-RELEASE r354233
> >> GENERIC  amd64
> >> and VirtIO support.
> >>
> >> I performed:
> >>
> >> vale-ctl -h vale0:vtnet1
> >> vale-ctl -n vi0
> >> vale-ctl -a vale0:vi0
> >>
> >> 615.925514 bdg_ctl [149] bridge:0 port:0 vale0:vtnet1
> >> 615.925559 bdg_ctl [149] bridge:0 port:1 vale0:vtnet1^
> >> 615.925572 bdg_ctl [149] bridge:0 port:2 vale0:vi0
> >>
> >> vtnet1 is configured as mirror port.
> >> But if:
> >>
> >> tcpdump -i vtnet1
> >> or
> >> tcpdump -i vale0:vi0
> >>
> >> why can't I see any TCP packets?
> >> UDP and ICMP packet are ok.
> >>
> >> Without vale switch tcpdump show all TCP packets correctly.
> >
> >
> > You have to disable checksum offloading on vtnet1.
> >
> > Cheers
> > Luigi
> >
> > It is a bug?
> >> Thanks to all!
> >> 
> >> ___
> >> freebsd-net@freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
> >>
> >
> >
> > --
> > -+---
> >  Prof. Luigi RIZZO, ri...@iet.unipi.it  . Dip. di Ing. dell'Informazione
> >  http://www.iet.unipi.it/~luigi/. Universita` di Pisa
> >  TEL  +39-050-2217533   . via Diotisalvi 2
> >  Mobile   +39-338-6809875   . 56122 PISA (Italy)
> > -+---
> >
> >
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap - Vale switch - tcp problem

[Anthony Arnaud]

I already disabled the checksum, the vtnet config is:

ifconfig vtnet1 -txcsum -rxcsum -tso4 -tso6 -lro -txcsum6 -rxcsum6 -vlanmtu
-vlanhwtag -vlanhwfilter -vlanhwtso -vlanhwcsum up promisc

vtnet1:
flags=28943 metric
0 mtu 1500
options=1800a8
ether 0e:bd:ec:7a:08:06
media: Ethernet 10Gbase-T 
status: active
nd6 options=29

Sorry for not having posted vtnet config before.
PS: VLAN_HWCSUM is not switchable off for some reason!
the problem is not that.

Cheers
Anthony




Il giorno gio 28 mag 2020 alle ore 16:05 Luigi Rizzo 
ha scritto:

>
>
> On Thursday, May 28, 2020, Anthony Arnaud 
> wrote:
>
>> Hi everyone!
>> I would like to create a vale switch with an interface attached with the
>> host stack and some virtual.
>> My env is a VM with FBSD-12.1 12.1-RELEASE FreeBSD 12.1-RELEASE r354233
>> GENERIC  amd64
>> and VirtIO support.
>>
>> I performed:
>>
>> vale-ctl -h vale0:vtnet1
>> vale-ctl -n vi0
>> vale-ctl -a vale0:vi0
>>
>> 615.925514 bdg_ctl [149] bridge:0 port:0 vale0:vtnet1
>> 615.925559 bdg_ctl [149] bridge:0 port:1 vale0:vtnet1^
>> 615.925572 bdg_ctl [149] bridge:0 port:2 vale0:vi0
>>
>> vtnet1 is configured as mirror port.
>> But if:
>>
>> tcpdump -i vtnet1
>> or
>> tcpdump -i vale0:vi0
>>
>> why can't I see any TCP packets?
>> UDP and ICMP packet are ok.
>>
>> Without vale switch tcpdump show all TCP packets correctly.
>
>
> You have to disable checksum offloading on vtnet1.
>
> Cheers
> Luigi
>
> It is a bug?
>> Thanks to all!
>> 
>> ___
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>
>
>
> --
> -+---
>  Prof. Luigi RIZZO, ri...@iet.unipi.it  . Dip. di Ing. dell'Informazione
>  http://www.iet.unipi.it/~luigi/. Universita` di Pisa
>  TEL  +39-050-2217533   . via Diotisalvi 2
>  Mobile   +39-338-6809875   . 56122 PISA (Italy)
> -+---
>
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap - Vale switch - tcp problem

[Luigi Rizzo]

On Thursday, May 28, 2020, Anthony Arnaud 
wrote:

> Hi everyone!
> I would like to create a vale switch with an interface attached with the
> host stack and some virtual.
> My env is a VM with FBSD-12.1 12.1-RELEASE FreeBSD 12.1-RELEASE r354233
> GENERIC  amd64
> and VirtIO support.
>
> I performed:
>
> vale-ctl -h vale0:vtnet1
> vale-ctl -n vi0
> vale-ctl -a vale0:vi0
>
> 615.925514 bdg_ctl [149] bridge:0 port:0 vale0:vtnet1
> 615.925559 bdg_ctl [149] bridge:0 port:1 vale0:vtnet1^
> 615.925572 bdg_ctl [149] bridge:0 port:2 vale0:vi0
>
> vtnet1 is configured as mirror port.
> But if:
>
> tcpdump -i vtnet1
> or
> tcpdump -i vale0:vi0
>
> why can't I see any TCP packets?
> UDP and ICMP packet are ok.
>
> Without vale switch tcpdump show all TCP packets correctly.


You have to disable checksum offloading on vtnet1.

Cheers
Luigi

It is a bug?
> Thanks to all!
> 
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>


-- 
-+---
 Prof. Luigi RIZZO, ri...@iet.unipi.it  . Dip. di Ing. dell'Informazione
 http://www.iet.unipi.it/~luigi/. Universita` di Pisa
 TEL  +39-050-2217533   . via Diotisalvi 2
 Mobile   +39-338-6809875   . 56122 PISA (Italy)
-+---
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/ixl and crc addition..

[Vincenzo Maffione]

Il giorno gio 26 mar 2020 alle ore 11:39 Alexandre Snarskii <
s...@snar.spb.ru> ha scritto:

> On Wed, Mar 25, 2020 at 11:31:30PM +0100, Vincenzo Maffione wrote:
> > Hi Alexandre,
> >   Thanks. Your patch looks good to me. I assume you have tested it?
>
> Sure, this patch is already handles production traffic.
>
> > I will commit that to stable/11.
>
> Thanks.
>

> >
> > The issue you report on stable/12 is more worrisome. The 'no space in TX
> ring'
> > condition (head==cur==tail) is ok per-se: on a subsequent poll() wakeup
> (e.g.
> > TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see tail moving
> forward,
> > therefore freeing some space to be used in the ring.
>
> Unfortunately, further NIOCTXSYNC did not freed queue.
>
> Let me explain my observations in details: this issue was reproduceable
> even under really light load (say, hundreds pps per queue). As the load
> was ligth, I was able to insert debugging output before and after any
> calls
> to ioctl and poll, code is like:
>
> while(1) {
> if (nm_tx_pending(txring)) {
> printf("%s: pre-txsync cur: %d, head: %d, tail: %d\n",
> thread_name, txring->cur, txring->head,
> txring->tail);
> if (ioctl(pa->fd, NIOCTXSYNC) == -1) {
> log event
> };
> printf("%s: post-txsync cur: %d, haad: %d, tail: %d\n",
> thread_name, txring->cur, txring->head,
> txring->tail);
> };
> // setup poll descriptors with POLLIN only
> printf("%s: pre-poll cur: %d, head: %d, tail: %d\n",
> thread_name, txring->cur, txring->head, txring->tail);
> ret = poll(fds, 2, 1000);
> printf("%s: post-poll cur: %d, head: %d, tail: %d\n",
> thread_name, txring->cur, txring->head, txring->tail);
>
>
Ok, but I have some preliminary questions and notes:
 - You show the code that syncs the tx ring to collect completed
transmissions (and thus advance tail). However, I do not see the code that
submits new packets to be transmitted (and thus advances head/tail). Where
is it? If this is happening in a separate thread, that leads to undefined
behaviour, e.g. ring resets (although you can't crash the kernel).


> Logs in normal situation (only single thread shown):
>
> Notice:nz(ixl0:2): pre-txsync cur: 649, head: 649, tail: 647
> Notice:nz(ixl0:2): post-txsync cur: 649, head: 649, tail: 648
> Notice:nz(ixl0:2): pre-poll cur: 649, head: 649, tail: 648
> Notice:nz(ixl0:2): post-poll cur: 649, head: 649, tail: 648
>
> in pre-txsync: cur = head = tail + 2 (sometimes tail + 3), queue
> is nearly empty.
> in post-txsync: cur = head = tail + 1, queue is flushed.
>
> however, after some time, txcsync call advanced tail not to head - 1,
> but to head:
>
> Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 652
> Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
>

I see. This is wrong and must not happen. Once you get here, the tx ring
becomes unusable, so we may as well ignore what happens afterwards.

>
> and following txcsyncs was not able to free (already empty) queue:
>
> Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
> Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
>
> application entered busy wait.. (in my case most packets just zero-copied
> from rxring to txring, so if there are no space in txring, packets are not
> consumed from rx).
>
> > However, the ring_reinit means that something is going wrong: either your
> > application is using the TX ring incorrectly, or there is a bug in the
> netmap
>
> ring_reinit was indeed caused by my incorrect use of txring: I
> tried to ignore situation head == tail and inject packet anyway..
>

Yes, ring reinit actually resets the ring variables to a sane value.

In case your issue is not caused by unsafe access to the tx ring from
multiple threads (see above), it would help to
know what value do you have in sysctl hw.ixl.enable_head_writeback, and
also try if the problem is still
there if you enable or disable that parameter (which is relevant for how
tail is advanced).


>
> > iflib code. Since FreeBSD 12, netmap support is provided by iflib, while
> before
> > netmap support was provided directly by the ixl driver.
> > In any case, it would probably help if you could provide some more
> 

Re: netmap/ixl and crc addition..

[Alexandre Snarskii]

On Wed, Mar 25, 2020 at 11:31:30PM +0100, Vincenzo Maffione wrote:
> Hi Alexandre,
>   Thanks. Your patch looks good to me. I assume you have tested it?

Sure, this patch is already handles production traffic.

> I will commit that to stable/11.

Thanks.

> 
> The issue you report on stable/12 is more worrisome. The 'no space in TX ring'
> condition (head==cur==tail) is ok per-se: on a subsequent poll() wakeup (e.g.
> TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see tail moving 
> forward,
> therefore freeing some space to be used in the ring.

Unfortunately, further NIOCTXSYNC did not freed queue. 

Let me explain my observations in details: this issue was reproduceable 
even under really light load (say, hundreds pps per queue). As the load 
was ligth, I was able to insert debugging output before and after any calls 
to ioctl and poll, code is like:

while(1) {
if (nm_tx_pending(txring)) {
printf("%s: pre-txsync cur: %d, head: %d, tail: %d\n",
thread_name, txring->cur, txring->head, txring->tail);
if (ioctl(pa->fd, NIOCTXSYNC) == -1) {
log event
};
printf("%s: post-txsync cur: %d, haad: %d, tail: %d\n",
thread_name, txring->cur, txring->head, txring->tail);
};
// setup poll descriptors with POLLIN only
printf("%s: pre-poll cur: %d, head: %d, tail: %d\n",
thread_name, txring->cur, txring->head, txring->tail);
ret = poll(fds, 2, 1000);
printf("%s: post-poll cur: %d, head: %d, tail: %d\n",
thread_name, txring->cur, txring->head, txring->tail);

Logs in normal situation (only single thread shown):

Notice:nz(ixl0:2): pre-txsync cur: 649, head: 649, tail: 647
Notice:nz(ixl0:2): post-txsync cur: 649, head: 649, tail: 648
Notice:nz(ixl0:2): pre-poll cur: 649, head: 649, tail: 648
Notice:nz(ixl0:2): post-poll cur: 649, head: 649, tail: 648

in pre-txsync: cur = head = tail + 2 (sometimes tail + 3), queue
is nearly empty.
in post-txsync: cur = head = tail + 1, queue is flushed.

however, after some time, txcsync call advanced tail not to head - 1, 
but to head:

Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 652
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654

and following txcsyncs was not able to free (already empty) queue:

Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-poll cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): pre-txsync cur: 654, head: 654, tail: 654
Notice:nz(ixl0:2): post-txsync cur: 654, head: 654, tail: 654

application entered busy wait.. (in my case most packets just zero-copied
from rxring to txring, so if there are no space in txring, packets are not
consumed from rx).

> However, the ring_reinit means that something is going wrong: either your
> application is using the TX ring incorrectly, or there is a bug in the netmap

ring_reinit was indeed caused by my incorrect use of txring: I 
tried to ignore situation head == tail and inject packet anyway.. 

> iflib code. Since FreeBSD 12, netmap support is provided by iflib, while 
> before
> netmap support was provided directly by the ixl driver.
> In any case, it would probably help if you could provide some more detailed
> info (how to reproduce the problem).
> 
> Cheers,
>   Vincenzo
> 
> Il giorno mar 24 mar 2020 alle ore 15:12 Alexandre Snarskii 
> ha scritto:
> 
> On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote:
> >
> > Hi!
> >
> > Long story short: looks like intel x722 does not by default add CRC to
> > outbound frames, so with FreeBSD 11-stable netmap-generated traffic is
> > dropped on the next port.. Fix is simple, attached.
> 
> ... add missing attach :(
> 
> >
> > The same behaviour of 'unconditionally ask card to compute crc' can
> > be found in both if_ixl:
> > https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=
> markup#l408
> > and in DPDK i40e driver:
> > https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#
> L1105
> > so, I guess, it's safe.
> >
> > PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while
> > this upgrade solved the crc problem, this version shows 'stalled tx
> queue'
> > problem: after CTXSYNC tail == head == cur, 'no space in ring' 
> condition.
> > Attempts to ignore this condition 

Re: netmap/ixl and crc addition..

[Slawa Olhovchenkov]

On Wed, Mar 25, 2020 at 11:31:30PM +0100, Vincenzo Maffione wrote:

> Hi Alexandre,
>   Thanks. Your patch looks good to me. I assume you have tested it?
> I will commit that to stable/11.
> 
> The issue you report on stable/12 is more worrisome. The 'no space in TX
> ring' condition (head==cur==tail) is ok per-se: on a subsequent poll()
> wakeup (e.g. TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see
> tail moving forward, therefore freeing some space to be used in the ring.
> However, the ring_reinit means that something is going wrong: either your
> application is using the TX ring incorrectly, or there is a bug in the
> netmap iflib code. Since FreeBSD 12, netmap support is provided by iflib,
> while before netmap support was provided directly by the ixl driver.
> In any case, it would probably help if you could provide some more detailed
> info (how to reproduce the problem).

Just run pkt-gen for transmit. The conditions is guaranteed to occur
(for me, at least).
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/ixl and crc addition..

[Vincenzo Maffione]

Hi Alexandre,
  Thanks. Your patch looks good to me. I assume you have tested it?
I will commit that to stable/11.

The issue you report on stable/12 is more worrisome. The 'no space in TX
ring' condition (head==cur==tail) is ok per-se: on a subsequent poll()
wakeup (e.g. TX interrupt) or explicit ioctl(NIOCTXSYNC) you should see
tail moving forward, therefore freeing some space to be used in the ring.
However, the ring_reinit means that something is going wrong: either your
application is using the TX ring incorrectly, or there is a bug in the
netmap iflib code. Since FreeBSD 12, netmap support is provided by iflib,
while before netmap support was provided directly by the ixl driver.
In any case, it would probably help if you could provide some more detailed
info (how to reproduce the problem).

Cheers,
  Vincenzo

Il giorno mar 24 mar 2020 alle ore 15:12 Alexandre Snarskii <
s...@snar.spb.ru> ha scritto:

> On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote:
> >
> > Hi!
> >
> > Long story short: looks like intel x722 does not by default add CRC to
> > outbound frames, so with FreeBSD 11-stable netmap-generated traffic is
> > dropped on the next port.. Fix is simple, attached.
>
> ... add missing attach :(
>
> >
> > The same behaviour of 'unconditionally ask card to compute crc' can
> > be found in both if_ixl:
> >
> https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=markup#l408
> > and in DPDK i40e driver:
> >
> https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#L1105
> > so, I guess, it's safe.
> >
> > PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while
> > this upgrade solved the crc problem, this version shows 'stalled tx
> queue'
> > problem: after CTXSYNC tail == head == cur, 'no space in ring'
> condition.
> > Attempts to ignore this condition led to continuous ring resets in
> txcsync:
> >
> > Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologue
> ixl1 TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t
> 135 rh 135 rc 135 rt 135 hc 135 ht 135
> > Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit
> called for ixl1 TX3
> >
> > PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with
> > two onboard X722 ports (actually, four, but two of them are not
> > wired).
> >
> > CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU)
> >   Origin="GenuineIntel"  Id=0x50654  Family=0x6  Model=0x55  Stepping=4
> >
> > ixl0:  1.11.9-k> mem 0xfa00-0xfaff,0xfb018000-0xfb01 irq 11 at device
> 0.0 numa-domain 0 on pci12
> > ixl0: using 1024 tx descriptors and 1024 rx descriptors
> > ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0
> > ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C
> > ixl0: Using MSIX interrupts with 9 vectors
> > ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
> > ixl0: Ethernet address: 3c:2c:30:30:59:85
> > ixl0: SR-IOV ready
> > ixl0: netmap queues/slots: TX 8/1024, RX 8/1024
> >
> > ixl0@pci0:184:0:0:class=0x02 card=0x8086 chip=0x37d38086
> rev=0x04 hdr=0x00
> > vendor = 'Intel Corporation'
> > device = 'Ethernet Connection X722 for 10GbE SFP+'
> > class  = network
> > subclass   = ethernet
> >
> >
> > ___
> > freebsd-net@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/ixl and crc addition..

[Alexandre Snarskii]

On Tue, Mar 24, 2020 at 03:37:36PM +0300, Alexandre Snarskii wrote:
> 
> Hi!
> 
> Long story short: looks like intel x722 does not by default add CRC to 
> outbound frames, so with FreeBSD 11-stable netmap-generated traffic is 
> dropped on the next port.. Fix is simple, attached.

... add missing attach :(

> 
> The same behaviour of 'unconditionally ask card to compute crc' can
> be found in both if_ixl:
> https://svnweb.freebsd.org/base/stable/11/sys/dev/ixl/ixl_txrx.c?view=markup#l408
> and in DPDK i40e driver:
> https://github.com/DPDK/dpdk/blob/master/drivers/net/i40e/i40e_rxtx.c#L1105
> so, I guess, it's safe.
> 
> PS: of course, first idea was to upgrade to FreeBSD 12-stable, but while
> this upgrade solved the crc problem, this version shows 'stalled tx queue' 
> problem: after CTXSYNC tail == head == cur, 'no space in ring' condition. 
> Attempts to ignore this condition led to continuous ring resets in txcsync:
> 
> Mar 17 20:21:08 host kernel: 668.224836 [1679] nm_txsync_prologueixl1 
> TX3: fail 'head < kring->rhead || head > kring->rtail' h 136 c 136 t 135 rh 
> 135 rc 135 rt 135 hc 135 ht 135
> Mar 17 20:21:08 host kernel: 668.238300 [1787] netmap_ring_reinit
> called for ixl1 TX3
> 
> PPS: hardware details: Dell VEP4600, based on Xeon D-2100 with 
> two onboard X722 ports (actually, four, but two of them are not
> wired).
> 
> CPU: Intel(R) Xeon(R) D-2187NT CPU @ 2.00GHz (2000.06-MHz K8-class CPU)
>   Origin="GenuineIntel"  Id=0x50654  Family=0x6  Model=0x55  Stepping=4
> 
> ixl0:  
> mem 0xfa00-0xfaff,0xfb018000-0xfb01 irq 11 at device 0.0 
> numa-domain 0 on pci12
> ixl0: using 1024 tx descriptors and 1024 rx descriptors
> ixl0: fw 3.1.57069 api 1.5 nvm 3.33 etid 80001007 oem 1.263.0
> ixl0: PF-ID[0]: VFs 32, MSIX 129, VF MSIX 5, QPs 384, I2C
> ixl0: Using MSIX interrupts with 9 vectors
> ixl0: Allocating 8 queues for PF LAN VSI; 8 queues active
> ixl0: Ethernet address: 3c:2c:30:30:59:85
> ixl0: SR-IOV ready
> ixl0: netmap queues/slots: TX 8/1024, RX 8/1024
> 
> ixl0@pci0:184:0:0:class=0x02 card=0x8086 chip=0x37d38086 rev=0x04 
> hdr=0x00
> vendor = 'Intel Corporation'
> device = 'Ethernet Connection X722 for 10GbE SFP+'
> class  = network
> subclass   = ethernet
> 
> 
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Index: sys/dev/netmap/if_ixl_netmap.h
===
--- sys/dev/netmap/if_ixl_netmap.h	(revision 359243)
+++ sys/dev/netmap/if_ixl_netmap.h	(working copy)
@@ -239,7 +239,8 @@
 			curr->cmd_type_offset_bsz = htole64(
 			((u64)len << I40E_TXD_QW1_TX_BUF_SZ_SHIFT) |
 			flags |
-			((u64)I40E_TX_DESC_CMD_EOP << I40E_TXD_QW1_CMD_SHIFT)
+			((u64)I40E_TX_DESC_CMD_EOP << I40E_TXD_QW1_CMD_SHIFT) |
+			((u64)I40E_TX_DESC_CMD_ICRC << I40E_TXD_QW1_CMD_SHIFT)
 			  ); // XXX more ?
 
 			/* make sure changes to the buffer are synced */
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap on cxgb (Chelsio T3) — panic on transmit

[Navdeep Parhar]

On 11/22/18 7:30 AM, Lev Serebryakov wrote:
> 
>  I've obtained Chelsio T3 for my "network lab". It works with cxgb
> driver well, but when I try to use Netmap's pkt-gen on it it crashes
> system immediately with such message:
> 
> panic: trying to coalesce 9 packets in to one WR
> 
> I've turned all checksums, lro and tso off, but it doesn't help.
> 
> Do I have any chances to get netmap supported (maybe, not very
> efficient) on this NIC?
> 

The T3 is a very old chip that has been EoL'd for some time and it's not
likely to get native netmap support.

Your panic must be while using netmap's emulation mode on top of cxgb.
Try modifying check_pkt_coalesce() in the driver to always return 0 and
see if that avoids the panic.  Don't expect much performance-wise even
if that works.

Regards,
Navdeep
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap & chelsio

[Eggert, Lars]

Hi,

On 2018-7-11, at 18:53, Navdeep Parhar  wrote:
> Try changing lazy_tx_credit_flush to 0 on the running kernel with a
> debugger, or compile the driver with it set to 0 -- it's in t4_netmap.c:
> 
> int lazy_tx_credit_flush = 1;

thanks! With that, I get performance similar to the ixl cards on first try.

> I'm surprised I don't have a tunable/sysctl for it.  I'll add one really
> soon.

That would be useful!

(Also useful would be some definite documentation on what all the loader 
tunables are and what they do. But the current situation is already much better 
than for the Intel cards, where esp. those that have been iflib-ified seem to 
have completely undocumented tunables now.)

General/unrelated suggestion: Could the kernel spit out a warning when it 
encounters a loader tunable that doesn't do anything? That would allow one to 
at least catch when tunables are renamed/changed.

Lars


signature.asc
Description: Message signed with OpenPGP

Re: netmap & chelsio

[Navdeep Parhar]

On 07/11/18 07:58, Eggert, Lars wrote:
> Hi,
> 
> I have netmap working with the T6 cards now.
> 
> However, performance is very poor. It seems to take several milliseconds 
> after a NIOCTXSYNC ioctl before the tail is updated?

Try changing lazy_tx_credit_flush to 0 on the running kernel with a
debugger, or compile the driver with it set to 0 -- it's in t4_netmap.c:

int lazy_tx_credit_flush = 1;

I'm surprised I don't have a tunable/sysctl for it.  I'll add one really
soon.


Regards,
Navdeep

> 
> In case it matters, here is what is in loader.conf:
> 
> hw.cxgbe.num_vis=2
> hw.cxgbe.fl_pktshift=0
> hw.cxgbe.ntxq=1
> hw.cxgbe.nrxq=1
> hw.cxgbe.qsize_txq=512
> hw.cxgbe.qsize_rxq=512
> hw.cxgbe.cong_drop=1
> hw.cxgbe.pause_settings=1
> hw.cxgbe.autoneg=0
> hw.cxgbe.nm_rx_nframes=1
> hw.cxgbe.nm_rx_ndesc=1
> 
> Lars
> 

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap & chelsio

[Eggert, Lars]

Hi,

I have netmap working with the T6 cards now.

However, performance is very poor. It seems to take several milliseconds after 
a NIOCTXSYNC ioctl before the tail is updated?

In case it matters, here is what is in loader.conf:

hw.cxgbe.num_vis=2
hw.cxgbe.fl_pktshift=0
hw.cxgbe.ntxq=1
hw.cxgbe.nrxq=1
hw.cxgbe.qsize_txq=512
hw.cxgbe.qsize_rxq=512
hw.cxgbe.cong_drop=1
hw.cxgbe.pause_settings=1
hw.cxgbe.autoneg=0
hw.cxgbe.nm_rx_nframes=1
hw.cxgbe.nm_rx_ndesc=1

Lars


signature.asc
Description: Message signed with OpenPGP

Re: netmap & chelsio

[n...@freebsd.org]

On Fri, Jul 06, 2018 at 06:41:53AM +, Eggert, Lars wrote:
> Hi,
> 
> On 2018-7-5, at 17:47, n...@freebsd.org wrote:
> > Set hw.cxgbe.fl_pktshift=0 in loader.conf to stop the chip from doing
> > this.  See cxgbe(4) for details on the knob.  It's a historic
> > optimization that doesn't seem to matter on modern CPUs, so the driver
> > default should probably be 0 instead of 2.
> 
> thanks, I must have missed this in the man page.
> 
> Looking at this in detail now, I wonder if there are other loader
> settings that should be set for netmap use, such as
> hw.cxgbe.buffer_packing=0 and/or hw.cxgbe.allow_mbufs_in_cluster=0?

The driver uses dedicated tx/rx queues for netmap and those queues use
netmap provided buffers instead of driver allocated mbuf+cluster used by
normal rx queues.  buffer_packing and allow_mbufs_in_cluster are not
applicable to netmap queues.

Regards,
Navdeep
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap & chelsio

[Eggert, Lars]

Hi,

On 2018-7-5, at 17:47, n...@freebsd.org wrote:
> Set hw.cxgbe.fl_pktshift=0 in loader.conf to stop the chip from doing
> this.  See cxgbe(4) for details on the knob.  It's a historic
> optimization that doesn't seem to matter on modern CPUs, so the driver
> default should probably be 0 instead of 2.

thanks, I must have missed this in the man page.

Looking at this in detail now, I wonder if there are other loader settings that 
should be set for netmap use, such as hw.cxgbe.buffer_packing=0 and/or 
hw.cxgbe.allow_mbufs_in_cluster=0?

Thanks,
Lars


signature.asc
Description: Message signed with OpenPGP

Re: netmap & chelsio

[n...@freebsd.org]

On Thu, Jul 05, 2018 at 03:09:33PM +, Eggert, Lars wrote:
> Hi,
> 
> when receiving packets via netmap (current GitHub version) on a Chelsio T6
> vcc0 device on -CURRENT, it appears that the Ethernet header starts at an
> offset of two bytes into the netmap slot. So far, I have only used netmap with
> various Intel NICs, where the Ethernet header starts at offset zero.
> 
> Is this a bug? Is this to be expected?

Set hw.cxgbe.fl_pktshift=0 in loader.conf to stop the chip from doing
this.  See cxgbe(4) for details on the knob.  It's a historic
optimization that doesn't seem to matter on modern CPUs, so the driver
default should probably be 0 instead of 2.

Regards,
Navdeep
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap: How the buf_num of buffers is used by driver and monitor app

[Vincenzo Maffione]

Hi,


2018-03-30 6:00 GMT+02:00 Ming Fu :

> Hi,
>
> I am wondering about the netmap module parameters buf_num. The default
> buf_num 163840 seems fairly big compare to intel 10G ixgbe of max 4096
> queue size. A full queue of the interface can only use a very small portion
> of the buf_num. Can the netmap enabled driver like ixgbe use more buffers
> than the "ethtool -G" configured rx/tx queue size?
>

No, netmap will just the same number of buffers as configured with ethtool
-G. The default buf_num is just a default that covers many things that you
can do.
First of all you would need 4096 for each TX and RX ring (so typically
2*8*4096). Then you need additional 2*4096 buffers for the host rings.
If you create netmap pipes like 'netmap:eth0{1' you need more buffers for
the pipe TX and RX ring.
If you know how many netmap buffers you really need for your purpose you
can just lower buf_num to the minimun number that won't cause your
application to fail with ENOMEM.


>
> I need to feed packages captured from netmap device to a few traffic
> monitors on the same box. The monitor application attach to the device as
> netmap monitor with the /r at the end of device name. My question is if the
> primary read of the device calls poll(), the netmap buffer is synced with
> the kernel. Can the other monitor application still access the packet that
> the primary read just returned to the kernel? If one of the monitor on the
> netmap device is slow, will it cause trouble for the primary reader and
> other monitors? How can I cache a lot of packets in the buffer in case one
> of the monitor application had a temporary slow down?
>

If you use copy-based monitor (no 'z') each monitor will receive an
independent copy of the traffic going through monitored rings. So I think
if one of the monitoring applications slows down there should
be no effect on the others. If you use zerocopy monitor, the same
intercepted buffer is passed between the various monitoring applications
(and the main application), so I think if one
stops or slows down, then all the monitors will stop.
There is some explanation in the comment here
https://github.com/luigirizzo/netmap/blob/master/sys/dev/netmap/netmap_monitor.c#L30-L69
Feel free to open a ticket on github (
https://github.com/luigirizzo/netmap/issues) if you have a more specific
question.

Cheers,
  Vincenzo


>
> Thanks,
> Ming
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap on Linux nm_open() fail when receive ring size is set to 4096

[Vincenzo Maffione]

Hi,
  You need to increase the value of some netmap module parameter. At least
ring_size, maybe also buf_num. Then restart your netmap applications.

Keep in mind that performance could worsen with more slots, because of
increased cache thrashing.

Cheers,
  Vincenzo


On Wed, Mar 28, 2018, 3:50 PM Ming Fu  wrote:

> Hi,
>
> I was trying netmap on a Linux box with 128G of ram (64G per numa node).
> If I set ixgbe interface to 4096 ring size, the nm_open will fail with
> error "Cannot allocate memory". What can I tweak to make the card use
> larger ring size? The following test was run after fresh reboot.
>
> $ ethtool -g enp5s0f0
> Ring parameters for enp5s0f0:
> Pre-set maximums:
> RX:4096
> RX Mini:   0
> RX Jumbo:   0
> TX:4096
> Current hardware settings:
> RX:512
> RX Mini:   0
> RX Jumbo:   0
> TX:512
>
> $ ethtool -G enp5s0f0 rx 1024
> $ ./nmtest -i enp5s0f0
> ^C
> $ ethtool -G enp5s0f0 rx 2048
> $ nmtest -i enp5s0f0
> ^C
> $ ethtool -G enp5s0f0 rx 4096
> $ nmtest -i enp5s0f0
> 816.039684 nm_open [945] NIOCREGIF failed: Cannot allocate memory
> netmap:enp5s0f0
> fail to nm_open(netmap:enp5s0f0 ... ): Cannot allocate memory
>
> Thanks,
> Ming
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf mtu

[Joe Buehler]

Grazie mille. I will test it as time allows -- swamped.

Joe Buehler

Vincenzo Maffione wrote:
> Hi,
>   This commit (fe13476b106ed1f4b517b1590e1dfb3f268b6e78) in the upstream
> netmap should have fixed the NS_MOREFRAG issue for ixgbe.
> If you happen give a try let us know.
> 
> Cheers,
>   Vincenzo
> 
> 2018-03-21 21:40 GMT+01:00 Vincenzo Maffione  >:
> 
> I see. Unfortunately this breaks the API, so I don't think we can
> accept it.
> We should probably sum up the fragment lengths, remember which one
> was the first descriptor and write the olinfo field when we process
> the last descriptor ...
> I hope this does not slow down the simpler case where NS_MOREFRAG is
> not used.
> 
> In any, case we should move this discussion to the github, if
> possible (so that the issue gets tracked).
> 
> Cheers,
>   Vincenzo
> 
> 2018-03-20 20:54 GMT+01:00 Joe Buehler  >:
> 
> Attached is a patch that allows fragmented TX with the ixgbevf
> driver.
> 
> For the first TX buffer set the slot length to the full length
> of the frame and make sure that the slot buffer is fully
> filled.  For succeeding slots just set the length to the amount
> of the buffer filled.
> 
> Not intended as the perfect solution but it works fine for my
> situation.
> 
> Joe Buehler
> 
> 
> 
> 
> -- 
> Vincenzo Maffione
> 
> 
> 
> 
> -- 
> Vincenzo Maffione

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf mtu

[Vincenzo Maffione]

Hi,
  This commit (fe13476b106ed1f4b517b1590e1dfb3f268b6e78) in the upstream
netmap should have fixed the NS_MOREFRAG issue for ixgbe.
If you happen give a try let us know.

Cheers,
  Vincenzo

2018-03-21 21:40 GMT+01:00 Vincenzo Maffione :

> I see. Unfortunately this breaks the API, so I don't think we can accept
> it.
> We should probably sum up the fragment lengths, remember which one was the
> first descriptor and write the olinfo field when we process the last
> descriptor ...
> I hope this does not slow down the simpler case where NS_MOREFRAG is not
> used.
>
> In any, case we should move this discussion to the github, if possible (so
> that the issue gets tracked).
>
> Cheers,
>   Vincenzo
>
> 2018-03-20 20:54 GMT+01:00 Joe Buehler :
>
>> Attached is a patch that allows fragmented TX with the ixgbevf driver.
>>
>> For the first TX buffer set the slot length to the full length of the
>> frame and make sure that the slot buffer is fully filled.  For succeeding
>> slots just set the length to the amount of the buffer filled.
>>
>> Not intended as the perfect solution but it works fine for my situation.
>>
>> Joe Buehler
>>
>>
>
>
> --
> Vincenzo Maffione
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf max frame size

[John-Mark Gurney]

Joseph H. Buehler wrote this message on Wed, Mar 21, 2018 at 09:32 -0400:
> This turned out to be a limitation of the box we are using, there is 
> apparently hardware between the ixgbevf chip and the fiber.

I've written a program that will probe the sizes that work successfully,
set the route's MTU properly.  This is designed for FreeBSD, but should
be adaptable to Linux, assuming it has similar features:
https://github.com/jmgurney/automtud

> > I am unable to send frames larger than 9216 bytes (destination MAC through 
> > trailing CRC inclusive) using ixgbevf hardware with latest netmap code 
> > (LINUX).
> > 
> > What is the source of this limitation?  From the chip datasheet it appears 
> > that much larger frames are supported.
> > 
> > There is mention of 9216 in some of the driver source files but as an MTU, 
> > the max frame size is larger.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf mtu

[Vincenzo Maffione]

I see. Unfortunately this breaks the API, so I don't think we can accept it.
We should probably sum up the fragment lengths, remember which one was the
first descriptor and write the olinfo field when we process the last
descriptor ...
I hope this does not slow down the simpler case where NS_MOREFRAG is not
used.

In any, case we should move this discussion to the github, if possible (so
that the issue gets tracked).

Cheers,
  Vincenzo

2018-03-20 20:54 GMT+01:00 Joe Buehler :

> Attached is a patch that allows fragmented TX with the ixgbevf driver.
>
> For the first TX buffer set the slot length to the full length of the
> frame and make sure that the slot buffer is fully filled.  For succeeding
> slots just set the length to the amount of the buffer filled.
>
> Not intended as the perfect solution but it works fine for my situation.
>
> Joe Buehler
>
>


-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf max frame size

[Joseph H. Buehler]

This turned out to be a limitation of the box we are using, there is apparently 
hardware between the ixgbevf chip and the fiber.

Joe Buehler

> I am unable to send frames larger than 9216 bytes (destination MAC through 
> trailing CRC inclusive) using ixgbevf hardware with latest netmap code 
> (LINUX).
> 
> What is the source of this limitation?  From the chip datasheet it appears 
> that much larger frames are supported.
> 
> There is mention of 9216 in some of the driver source files but as an MTU, 
> the max frame size is larger.
> 
> Joe Buehler
> 
> 

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf mtu

[Joe Buehler]

Attached is a patch that allows fragmented TX with the ixgbevf driver.

For the first TX buffer set the slot length to the full length of the frame and 
make sure that the slot buffer is fully filled.  For succeeding slots just set 
the length to the amount of the buffer filled.

Not intended as the perfect solution but it works fine for my situation.

Joe Buehler

--- LINUX/ixgbe_netmap_linux.h.~1~	2018-03-19 12:36:09.1 -0400
+++ LINUX/ixgbe_netmap_linux.h	2018-03-20 14:54:47.1 -0400
@@ -329,7 +329,8 @@
 		nic_i = netmap_idx_k2n(kring, nm_i);
 		for (n = 0; nm_i != head; n++) {
 			struct netmap_slot *slot = >slot[nm_i];
-			u_int len = slot->len;
+			u_int full_len = slot->len;
+			u_int len = full_len > ring->nr_buf_size ? ring->nr_buf_size : full_len;
 			uint64_t paddr;
 			void *addr = PNMB(na, slot, );
 
@@ -347,8 +348,8 @@
 
 			/* Fill the slot in the NIC ring. */
 			curr->read.buffer_addr = htole64(paddr);
-			curr->read.olinfo_status = htole32(len << IXGBE_ADVTXD_PAYLEN_SHIFT);
-			curr->read.cmd_type_len = htole32(len | hw_flags |
+			curr->read.olinfo_status = htole32(full_len << IXGBE_ADVTXD_PAYLEN_SHIFT);
+			curr->read.cmd_type_len = htole32(len | hw_flags |
 IXGBE_ADVTXD_DTYP_DATA | IXGBE_ADVTXD_DCMD_DEXT |
 IXGBE_ADVTXD_DCMD_IFCS);
 			netmap_sync_map(na, (bus_dma_tag_t) na->pdev, , len, NR_TX);
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf mtu

[Joe Buehler]

If you could fix this I would appreciate it.  My L1 TX performance for 64 byte 
L2 frames drops significantly, from about 9 gigbits to 7 gigabits, when I use 
buffers large enough to hold jumbo frames.

I will play with it a bit locally.  One thing that occurs to me is to pass the 
full frame length when NS_MOREFRAG is set and have the driver assume that the 
buffer is full if that length is more than the buffer length.  That way netmap 
knows both the total length and the fragment length.

Very ugly but...

Joe Buehler

> I don't think so, but you need to use the latest netmap from github.
> The NS_MOREFRAG support for ixgbe/ixgbevf is here
> https://github.com/luigirizzo/netmap/blob/master/LINUX/ixgbe_netmap_linux.h#L344-L345
> 
> The problem is that nobody really tried to use NS_MOREFRAG on ixgbevf
> transmission so far.
> So there may be a bug on how we set the flags in the hardware descriptor.
> We should look at what the driver does. Here
> https://elixir.bootlin.com/linux/v3.8/source/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#L2903
> I see that the olinfo_status field is set with the total frame length
> (and not just the fragment length).
> In the netmap code we set to the fragment length, so that's probably why
> you see that behaviour.
> Here
> https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf
> in sec. 7.2.3.2.4 I read that we need to properly set the olinfo_status
> field on the firsts TX descriptor, while the others are irrelevant.
> 
> Cheers,
>   Vincenzo
> 
>  
> 
>  
> 
> Joe Buehler
> 
> 
> 
> 
> -- 
> Vincenzo Maffione

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf mtu

[Vincenzo Maffione]

2018-03-19 20:49 GMT+01:00 Joe Buehler :

> Vincenzo Maffione wrote:
>
> > To receive a frame larger than the RX buffer size you need multiple
> > netmap slots (as multiple descriptors are
> > used by the hardware), looking at the NS_MOREFRAG flag.
> > See the example code in utils/functional.c::rx_one().
>
> This works fine -- thanks.
>
> > Also TX may have per-slot limitations (e.g. due to the size of the NIC
> > TX fifo), but this is usually > 9K, so using a single descriptor per
> > packet should always
> > be ok. However, you can also use multiple slots on the TX side (see
> > utils/functional.c::tx_one()).
>
> Trying to split TX frames into multiple buffers does not work, the NIC is
> sending 2048 byte frames (the buf_size I am using).
>
> I will re-check my code.  Do I need a particular version of ixgbevf
> perhaps?
>
>
I don't think so, but you need to use the latest netmap from github.
The NS_MOREFRAG support for ixgbe/ixgbevf is here
https://github.com/luigirizzo/netmap/blob/master/LINUX/ixgbe_netmap_linux.h#L344-L345

The problem is that nobody really tried to use NS_MOREFRAG on ixgbevf
transmission so far.
So there may be a bug on how we set the flags in the hardware descriptor.
We should look at what the driver does. Here
https://elixir.bootlin.com/linux/v3.8/source/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#L2903
I see that the olinfo_status field is set with the total frame length (and
not just the fragment length).
In the netmap code we set to the fragment length, so that's probably why
you see that behaviour.
Here
https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf
in sec. 7.2.3.2.4 I read that we need to properly set the olinfo_status
field on the firsts TX descriptor, while the others are irrelevant.

Cheers,
  Vincenzo





> Joe Buehler
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf mtu

[Joe Buehler]

Vincenzo Maffione wrote:

> To receive a frame larger than the RX buffer size you need multiple
> netmap slots (as multiple descriptors are
> used by the hardware), looking at the NS_MOREFRAG flag.
> See the example code in utils/functional.c::rx_one().

This works fine -- thanks.

> Also TX may have per-slot limitations (e.g. due to the size of the NIC
> TX fifo), but this is usually > 9K, so using a single descriptor per
> packet should always
> be ok. However, you can also use multiple slots on the TX side (see
> utils/functional.c::tx_one()).

Trying to split TX frames into multiple buffers does not work, the NIC is 
sending 2048 byte frames (the buf_size I am using).

I will re-check my code.  Do I need a particular version of ixgbevf perhaps?

Joe Buehler
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf mtu

[Vincenzo Maffione]

First, please make sure to use the latest netmap from github.

Yes, drivers in general use 2K or 4K RX buffers regardless of the MTU or
netmap buffer size.
To receive a frame larger than the RX buffer size you need multiple netmap
slots (as multiple descriptors are
used by the hardware), looking at the NS_MOREFRAG flag.
See the example code in utils/functional.c::rx_one().
Also TX may have per-slot limitations (e.g. due to the size of the NIC TX
fifo), but this is usually > 9K, so using a single descriptor per packet
should always
be ok. However, you can also use multiple slots on the TX side (see
utils/functional.c::tx_one()).

You need to set the buf_size parameter to the RX buffer size.
Currently we miss a mechanism for netmap to get the actual RX buffer size
from the NIC driver, so we assume 4K.
We need to check that buf_size is >= RX buffer size. This mechanism will be
added soon.

Cheers,
  Vincenzo

2018-03-16 23:52 GMT+01:00 Joe Buehler :

> Sorry, I should have added, this is LINUX if it matters.
>
> Joe Buehler wrote:
> > I am having difficulties with netmap over top of ixgbevf when attempting
> to use a large MTU (say 9000 bytes).
> >
> > Does the ixgbevf driver use 2048 byte buffers for RX regardless of the
> MTU or netmap buffer size?
> >
> > I can send large frames just fine but inbound frames are passed via
> netmap as 2048 bytes max.  It is possible that netmap is passing frames in
> multiple pieces I suppose, I haven't checked that yet -- my code is looking
> at the frame headers only at the moment so would toss trailing pieces.
> >
> > Joe Buehler
> >
> >
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap ixgbevf mtu

[Joe Buehler]

Sorry, I should have added, this is LINUX if it matters.

Joe Buehler wrote:
> I am having difficulties with netmap over top of ixgbevf when attempting to 
> use a large MTU (say 9000 bytes).
> 
> Does the ixgbevf driver use 2048 byte buffers for RX regardless of the MTU or 
> netmap buffer size?
> 
> I can send large frames just fine but inbound frames are passed via netmap as 
> 2048 bytes max.  It is possible that netmap is passing frames in multiple 
> pieces I suppose, I haven't checked that yet -- my code is looking at the 
> frame headers only at the moment so would toss trailing pieces.
> 
> Joe Buehler
> 
> 

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap memory config query

[Vincenzo Maffione]

Hi,
  The parameter to increase is "ring_size". FreeBSD or Linux does not
matter (except for how you modify the parameter, i.e. sysctl on FreeBSD and
sysfs in Linux).

Cheers,
  Vincenzo

2018-03-14 20:59 GMT+01:00 Joe Buehler :

> When I increase the ixgbevf rx ring size from 1024 to 4096 I get this:
>
> [ 5526.651780] 851.135009 [ 931] netmap_obj_malloc netmap_ring
> request size 65792 too large
> [ 5526.652779] 851.136008 [1802] netmap_mem2_rings_create  Cannot allocate
> RX_ring
>
> What netmap module parameters do I need to tweak to get past this?
>
> I am using LINUX should it matter.
>
> Joe Buehler
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap: Build a network SPAN/TAP from netmap

[Vincenzo Maffione]

Yes, or if you prefer you can simply extend "bridge" forwarding logic to
copy every packet to an additional TAP port. Copying a packet in netmap is
just a matter of initialize the next struct netmap_slot in the destination
(TAP) netmap ring, memcpy() the packet payload and incrementing
ring->cur/ring->head (then you need to TXSYNC or poll() at the end of the
batch.
In any case the application will work on both FreeBSD and Linux, as the API
is the same.

You may also find useful to look at the netmap tutorial, to see more
examples and explanations: https://github.com/vmaffione/netmap-tutorial

Cheers,
  Vincenzo

2017-12-15 6:58 GMT+01:00 Jim Thompson :

>
>
> > On Dec 14, 2017, at 12:00 PM, Ming Fu  wrote:
> >
> > Hi,
> >
> > I am trying to explore the possibility to build a network SPAN/TAP from
> netmap. Similar to the bridge sample, but all packet going through the
> bridge also get copied to a SPAN port. How do I duplicate or clone an
> incoming packet and send the original to bridge peer and the cloned one to
> the SPAN port? Is there an API like FreeBsd m_copypacket() for netmap?
> Would it work for Linux as well?
> >
> > Thanks
> > Ming
>
> Ming,
>
> I’d look at adapting netmap monitors.
>
> https://github.com/luigirizzo/netmap/blob/master/sys/dev/
> netmap/netmap_monitor.c
>
> For the rest of the solution, look at netmap_user.h, where it explains how
> to open a port in monitor mode.
>
> https://github.com/luigirizzo/netmap/blob/master/sys/net/netmap_user.h
>
> Essentially, once you have an active netmap port e.g. netmap:ix0, you can
> sniff the traffic by opening additional netmap ports
> named netmap:ix0/r (for rx traffic) or netmap:ix0/t (for tx) or even
> netmap:ix0/rt  (for both tx and rx)
>
> The rest of the code (to inject frames back down another interface) can be
> lifted from the bridge sample.
>
> You could also look at SF-TAP. http://sf-tap.github.io
>
> Jim
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap: Build a network SPAN/TAP from netmap

[Jim Thompson]

> On Dec 14, 2017, at 12:00 PM, Ming Fu  wrote:
> 
> Hi,
> 
> I am trying to explore the possibility to build a network SPAN/TAP from 
> netmap. Similar to the bridge sample, but all packet going through the bridge 
> also get copied to a SPAN port. How do I duplicate or clone an incoming 
> packet and send the original to bridge peer and the cloned one to the SPAN 
> port? Is there an API like FreeBsd m_copypacket() for netmap? Would it work 
> for Linux as well?
> 
> Thanks
> Ming

Ming,

I’d look at adapting netmap monitors.


https://github.com/luigirizzo/netmap/blob/master/sys/dev/netmap/netmap_monitor.c

For the rest of the solution, look at netmap_user.h, where it explains how to 
open a port in monitor mode.

https://github.com/luigirizzo/netmap/blob/master/sys/net/netmap_user.h

Essentially, once you have an active netmap port e.g. netmap:ix0, you can sniff 
the traffic by opening additional netmap ports
named netmap:ix0/r (for rx traffic) or netmap:ix0/t (for tx) or even 
netmap:ix0/rt  (for both tx and rx)

The rest of the code (to inject frames back down another interface) can be 
lifted from the bridge sample.

You could also look at SF-TAP. http://sf-tap.github.io

Jim

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap ouch double free

[Martina Balintova]

Hi Vincenzo,

Thank you for pointing me in right direction. Yes indeed, those were
overflow buffers that did the trick. I did not point them back before
closing.
(I am using linux).

Martina



On Dec 3, 2017 08:56, "Vincenzo Maffione"  wrote:

> Hi,
>   It may be related to the "extra buffers feature" of netmap, that lb uses.
> When the netmap port is opened, some additional buffers (not bound to any
> netmap ring) are allocated to be used by the application for slot swapping.
> They are provided through the ni_bufs_head field as a linked list (see
> updated netmap manual https://github.com/luigirizzo/
> netmap/blob/master/share/man/man4/netmap.4 ).
> To free them, the free list must be returned when closing the netmap file
> descriptor, using the same ni_bufs_head field.
>
> To check if this is true you could make sure that you are using "-B 0"
> option, which means no extra buffers are used.
>
> Are you using the latest netmap code from github? Are you using FreeBSD or
> Linux?
>
> Cheers,
>   Vincenzo
>
> 2017-12-01 15:21 GMT+01:00 Martina Balintova :
>
>> Hi,
>> I am currently playing with lb app in netmap. Every time I kill/close it,
>> the app hangs for some time and does not end immediately. In syslogs I am
>> getting:
>> '
>> Netmap_do_unregif   deleting last instance for myapp{1
>> Netmap_do_deref active=5
>> Netnap_obj_free   ouch, double free on buffer 2
>> Netmap_extra_free   freed 0 buffers
>> '
>>
>> This happens when I have some consumer on the pipe and at some point
>> during
>> whole lb lifetime, it did not consume all packets (resulting in oq being
>> filled or packets being dropped). If the pipe did not ever have a
>> consumer,
>> then it will not end up in the double free.
>> I am finding it quite hard to debug in gdb, as this is happening at the
>> shutdown.
>> Could someone pls point me to reason?
>>
>> I am running lb with one interface and 2 groups, different numb of pipes
>> per group and this happens even with no  extra buffers.
>>
>> Martina
>> ___
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>
>
>
>
> --
> Vincenzo Maffione
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap ouch double free

[Vincenzo Maffione]

Hi,
  It may be related to the "extra buffers feature" of netmap, that lb uses.
When the netmap port is opened, some additional buffers (not bound to any
netmap ring) are allocated to be used by the application for slot swapping.
They are provided through the ni_bufs_head field as a linked list (see
updated netmap manual
https://github.com/luigirizzo/netmap/blob/master/share/man/man4/netmap.4 ).
To free them, the free list must be returned when closing the netmap file
descriptor, using the same ni_bufs_head field.

To check if this is true you could make sure that you are using "-B 0"
option, which means no extra buffers are used.

Are you using the latest netmap code from github? Are you using FreeBSD or
Linux?

Cheers,
  Vincenzo

2017-12-01 15:21 GMT+01:00 Martina Balintova :

> Hi,
> I am currently playing with lb app in netmap. Every time I kill/close it,
> the app hangs for some time and does not end immediately. In syslogs I am
> getting:
> '
> Netmap_do_unregif   deleting last instance for myapp{1
> Netmap_do_deref active=5
> Netnap_obj_free   ouch, double free on buffer 2
> Netmap_extra_free   freed 0 buffers
> '
>
> This happens when I have some consumer on the pipe and at some point during
> whole lb lifetime, it did not consume all packets (resulting in oq being
> filled or packets being dropped). If the pipe did not ever have a consumer,
> then it will not end up in the double free.
> I am finding it quite hard to debug in gdb, as this is happening at the
> shutdown.
> Could someone pls point me to reason?
>
> I am running lb with one interface and 2 groups, different numb of pipes
> per group and this happens even with no  extra buffers.
>
> Martina
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap / LINUX realtime / ixgbevf: huge RX latencies

[Vincenzo Maffione]

HEAD/11 is the FreeBSD version.
Your question is about netmap on Linux, so this is the wrong mailing list.
You should open an issue on the official netmap
https://github.com/luigirizzo/netmap/issues

Cheers
  Vincenzo

2017-12-02 13:20 GMT+01:00 Joe Buehler :

> K. Macy wrote:
>
> > HEAD or 11?
>
> I'm not quite sure what the question means but there is this in the
> netmap code:
>
> ./net/netmap.h:42:#define   NETMAP_API  11  /* current
> API version */
>
> LINUX kernels nowadays can timestamp frames when they arrive from the
> NIC.  I made a trivial patch to the netmap driver to turn this on and
> also pass the timestamp to user space, and will pass on the changes.  I
> am doing frame latency measurements and this simple change eliminated a
> *whole* lot of noise in the measurements.
>
> Joe Buehler
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap / LINUX realtime / ixgbevf: huge RX latencies

[Joe Buehler]

K. Macy wrote:

> HEAD or 11?

I'm not quite sure what the question means but there is this in the
netmap code:

./net/netmap.h:42:#define   NETMAP_API  11  /* current API 
version */

LINUX kernels nowadays can timestamp frames when they arrive from the
NIC.  I made a trivial patch to the netmap driver to turn this on and
also pass the timestamp to user space, and will pass on the changes.  I
am doing frame latency measurements and this simple change eliminated a
*whole* lot of noise in the measurements.

Joe Buehler
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap / LINUX realtime / ixgbevf: huge RX latencies

[K. Macy]

HEAD or 11?


On Thu, Nov 30, 2017 at 13:03 Joe Buehler  wrote:

> I am using the LINUX 4.4.86 realtime kernel patch with netmap and the
> ixgbevf driver (SRIOV in a VM) and having some serious RX latency issues.
>
> The ixbgevf driver built by the netmap build against the kernel source
> does not work.  I haven't tried to debug it -- the netmap driver
> complains about bad ring index values and resets (over and over again as
> my app runs).
>
> The non-netmap version of ixgbevf that comes with the kernel works but
> appears to have serious RX latency issues due to frame batching.  It
> appears that the kernel's NAPI polling reads frames from the NIC, then
> netmap processes the frames, then epoll exits and my app can work on the
> frames.  The delay between start of NAPI to my app getting frames can be
> milliseconds!!!
>
> Would the netmap-modified ixgbevf driver eliminate this latency?
>
> If so, any ideas on what might be going wrong with the modified driver?
>  Perhaps an older version of the driver might be better?  The kernel
> ixgbevf version is 2.12.1-k while netmap is trying to use 4.2.1.
> Looking at the RT patch for LINUX there are only a few network driver
> patches, none for the ixgbe* cards.
>
> Joe Buehler
>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap / LINUX realtime / ixgbevf: huge RX latencies

[Joe Buehler]

Here is what I start seeing in dmesg when I attempt to use the 4.2.1
ixgbevf driver as modified for netmap.

Joe Buehler

[  136.607019] 343.712201 [ 189] ixgbe_netmap_configure_srrctl not supported
[  136.610020] 343.715203 [ 189] ixgbe_netmap_configure_srrctl not supported
[  136.796698] ixgbevf :00:08.0: NIC Link is Up 10 Gbps
[  136.953013] 344.058196 [ 189] ixgbe_netmap_configure_srrctl not supported
[  136.955014] 344.060198 [ 189] ixgbe_netmap_configure_srrctl not supported
[  136.997559] ixgbevf :00:08.0: NIC Link is Up 10 Gbps
[  136.997908] 344.103090 [1653] nm_rxsync_prologueeth0 RX0:
fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 1 c 1 t 0 rh
1 rc 1 rt 0 hc 0 ht 0
[  136.999652] 344.104836 [1698] netmap_ring_reinitcalled for
eth0 RX0
[  137.000568] 344.105751 [1653] nm_rxsync_prologueeth0 RX0:
fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 1 c 1 t 0 rh
1 rc 1 rt 0 hc 0 ht 0
[  137.002268] 344.107452 [1698] netmap_ring_reinitcalled for
eth0 RX0
[  137.003165] 344.108349 [1653] nm_rxsync_prologueeth0 RX0:
fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 1 c 1 t 0 rh
1 rc 1 rt 0 hc 0 ht 0
[  137.004856] 344.110041 [1698] netmap_ring_reinitcalled for
eth0 RX0
[  137.005752] 344.110936 [1653] nm_rxsync_prologueeth0 RX0:
fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 1 c 1 t 0 rh
1 rc 1 rt 0 hc 0 ht 0
[  137.007451] 344.112635 [1698] netmap_ring_reinitcalled for
eth0 RX0
[  137.008343] 344.113527 [1653] nm_rxsync_prologueeth0 RX0:
fail 'head < kring->nr_hwcur || head > kring->nr_hwtail' h 1 c 1 t 0 rh
1 rc 1 rt 0 hc 0 ht 0
[  137.010066] 344.115250 [1698] netmap_ring_reinitcalled for
eth0 RX0
[  137.010953] 344.116137 [1698] netmap_ring_reinitcalled for
eth0 RX0

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/vale periodic deadlock

[Vincenzo Maffione]

Hi,
  $ bridge -h
for usage.

Yes, bridge is just a simple example program, and stays in foreground.
Probably using daemon(8) you can turn it into a daemon.

Cheers,
  Vincenzo

2017-11-22 10:18 GMT+01:00 Harry Schmalzbauer :

>  Bezüglich Harry Schmalzbauer's Nachricht vom 22.11.2017 09:39 (localtime):
> > Bezüglich Vincenzo Maffione's Nachricht vom 22.11.2017 09:04 (localtime):
> >>
> >> 2017-11-21 21:48 GMT+01:00 Harry Schmalzbauer  >> >:
> >>
> >> Bezüglich Vincenzo Maffione's Nachricht vom 21.11.2017 09:39
> >> (localtime):
> >> …
> >> >
> >> > If this is the case, although you are allowed to do that, I don't
> think
> >> > it's a convenient way to use netmap.
> >> > Since VLAN interfaces like vlan0 do not have (and cannot have)
> native
> >> > netmap support, you are falling back to emulated netmap adapters
> (which
> >> > are probably buggy on FreeBSD, specially when combined with VALE).
> >> > Apart from bugs I think that with this setup you can't get decent
> >> > performance that would justify using netmap rather than the
> standard
> >> > kernel bridge and TAP devices.
> >>
> >> Hello,
> >>
> >> lockup happened earlier than expected.
> >> This time 'vale-ctl' still reported (-l) the configuration.
> >> One guest, using if_vtnet(4)-virtio-net#vale2:korso, showed:
> >> dmz: watchdog timeout on queue 0
> >> (dmz is the renamed if_vtnet(4))
> >>
> >> I could attach tcpdump to the uplink interface and also to all vlan
> >> children.
> >> Complete silence everywhere.  So it seems the nic stopped processing
> >> anything.
> >>
> >> Do you think that symptom could be caused by my special vale
> >> integration, so that bugs in netmap emulation could crash the NIC?
> >> Or is it unlikely that this is related.
> >>
> >> I hadn't prepared a debug kernel for the host, so the machine
> rebooted
> >> without again.
> >> I think I'll have to start with replacing vale first, to narrow down
> >> possible causes.  Today I was lucky, the lockup happend after
> business
> >> hours, but I won't rely on that.
> >> At least I know if I really need to look for a debug netmap kernel,
> or
> >> possibly there's something else...
> >>
> >> Thanks,
> >>
> >> -harry
> >>
> >>
> >>
> >> I can't really say anything without a stack trace or meaningful logs.
> >> There is a thing that you may do to see if the bug comes out of a bad
> >> interaction between
> >> emulated netmap and VALE.
> >> Instead of attaching the vlan interfaces to VALE you can connect VALE to
> >> the vlan interface
> >> through the "bridge" program. In this way nothing changes from the
> >> functional point of view,
> >> but you are not attaching anymore the VLAN interface to VALE (and you
> >> are using an additional process).
> >>
> >> So instead of
> >>
> >>   # vale-ctl vale0:vlan0
> >>
> >> you would have
> >>
> >>   # bridge netmap:vlan0 vale0:vv  # "vv" can be anything
> > Hello Vincenzo,
> >
> > thank you very much for that interesting hint.
> > I prepared a netgraph setup yesterday evening, but I'll try your
> > suggestion first.  Unfortunately I don't have time to prepare a debug
>
> Since this doesn't need a reboot and I'm in adventure mood, I just tried
> it at runtime.
> Unfortunately I can't find bridge documentation besides the source code.
> It doesn't detach from terminal here:
> bridge built Oct  8 2017
> 12:59:57
>
> 060.359974 main [244] --- zerocopy NOT
> supported
>
> 060.359987 main [251] Wait 4 secs for link to come up...
> 064.365872 main [255] Ready to go, nic1_egn 0x0/1 <-> vale4:nic1egn
> 0x0/1.
>
> 068.084364 main [306] poll timeout [0] ev 1 0 rx 0@33 tx 1022, [1] ev 1
> 0 rx 0@34 tx
> 1023
> 072.565559 main [306] poll timeout [0] ev 1 0 rx 0@34 tx 1022, [1] ev 1
> 0 rx 0@35 tx 1023
> …
>
> In general, things are working.
>
> Is bridge staing in the foreground by design?
>
> Thanks,
>
> -harry
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/vale periodic deadlock

[Harry Schmalzbauer]

 Bezüglich Harry Schmalzbauer's Nachricht vom 22.11.2017 09:39 (localtime):
> Bezüglich Vincenzo Maffione's Nachricht vom 22.11.2017 09:04 (localtime):
>>
>> 2017-11-21 21:48 GMT+01:00 Harry Schmalzbauer > >:
>>
>> Bezüglich Vincenzo Maffione's Nachricht vom 21.11.2017 09:39
>> (localtime):
>> …
>> >
>> > If this is the case, although you are allowed to do that, I don't think
>> > it's a convenient way to use netmap.
>> > Since VLAN interfaces like vlan0 do not have (and cannot have) native
>> > netmap support, you are falling back to emulated netmap adapters (which
>> > are probably buggy on FreeBSD, specially when combined with VALE).
>> > Apart from bugs I think that with this setup you can't get decent
>> > performance that would justify using netmap rather than the standard
>> > kernel bridge and TAP devices.
>>
>> Hello,
>>
>> lockup happened earlier than expected.
>> This time 'vale-ctl' still reported (-l) the configuration.
>> One guest, using if_vtnet(4)-virtio-net#vale2:korso, showed:
>> dmz: watchdog timeout on queue 0
>> (dmz is the renamed if_vtnet(4))
>>
>> I could attach tcpdump to the uplink interface and also to all vlan
>> children.
>> Complete silence everywhere.  So it seems the nic stopped processing
>> anything.
>>
>> Do you think that symptom could be caused by my special vale
>> integration, so that bugs in netmap emulation could crash the NIC?
>> Or is it unlikely that this is related.
>>
>> I hadn't prepared a debug kernel for the host, so the machine rebooted
>> without again.
>> I think I'll have to start with replacing vale first, to narrow down
>> possible causes.  Today I was lucky, the lockup happend after business
>> hours, but I won't rely on that.
>> At least I know if I really need to look for a debug netmap kernel, or
>> possibly there's something else...
>>
>> Thanks,
>>
>> -harry
>>
>>
>>
>> I can't really say anything without a stack trace or meaningful logs.
>> There is a thing that you may do to see if the bug comes out of a bad
>> interaction between
>> emulated netmap and VALE.
>> Instead of attaching the vlan interfaces to VALE you can connect VALE to
>> the vlan interface
>> through the "bridge" program. In this way nothing changes from the
>> functional point of view,
>> but you are not attaching anymore the VLAN interface to VALE (and you
>> are using an additional process).
>>
>> So instead of
>>
>>   # vale-ctl vale0:vlan0
>>
>> you would have
>>
>>   # bridge netmap:vlan0 vale0:vv  # "vv" can be anything
> Hello Vincenzo,
>
> thank you very much for that interesting hint.
> I prepared a netgraph setup yesterday evening, but I'll try your
> suggestion first.  Unfortunately I don't have time to prepare a debug

Since this doesn't need a reboot and I'm in adventure mood, I just tried
it at runtime.
Unfortunately I can't find bridge documentation besides the source code.
It doesn't detach from terminal here:
bridge built Oct  8 2017
12:59:57
  

060.359974 main [244] --- zerocopy NOT
supported   


060.359987 main [251] Wait 4 secs for link to come up...
064.365872 main [255] Ready to go, nic1_egn 0x0/1 <-> vale4:nic1egn
0x0/1.  
   

068.084364 main [306] poll timeout [0] ev 1 0 rx 0@33 tx 1022, [1] ev 1
0 rx 0@34 tx
1023  
072.565559 main [306] poll timeout [0] ev 1 0 rx 0@34 tx 1022, [1] ev 1
0 rx 0@35 tx 1023
…

In general, things are working.

Is bridge staing in the foreground by design?

Thanks,

-harry
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/vale periodic deadlock

[Harry Schmalzbauer]

Bezüglich Vincenzo Maffione's Nachricht vom 22.11.2017 09:04 (localtime):
> 
> 
> 2017-11-21 21:48 GMT+01:00 Harry Schmalzbauer  >:
> 
> Bezüglich Vincenzo Maffione's Nachricht vom 21.11.2017 09:39
> (localtime):
> …
> >
> > If this is the case, although you are allowed to do that, I don't think
> > it's a convenient way to use netmap.
> > Since VLAN interfaces like vlan0 do not have (and cannot have) native
> > netmap support, you are falling back to emulated netmap adapters (which
> > are probably buggy on FreeBSD, specially when combined with VALE).
> > Apart from bugs I think that with this setup you can't get decent
> > performance that would justify using netmap rather than the standard
> > kernel bridge and TAP devices.
> 
> Hello,
> 
> lockup happened earlier than expected.
> This time 'vale-ctl' still reported (-l) the configuration.
> One guest, using if_vtnet(4)-virtio-net#vale2:korso, showed:
> dmz: watchdog timeout on queue 0
> (dmz is the renamed if_vtnet(4))
> 
> I could attach tcpdump to the uplink interface and also to all vlan
> children.
> Complete silence everywhere.  So it seems the nic stopped processing
> anything.
> 
> Do you think that symptom could be caused by my special vale
> integration, so that bugs in netmap emulation could crash the NIC?
> Or is it unlikely that this is related.
> 
> I hadn't prepared a debug kernel for the host, so the machine rebooted
> without again.
> I think I'll have to start with replacing vale first, to narrow down
> possible causes.  Today I was lucky, the lockup happend after business
> hours, but I won't rely on that.
> At least I know if I really need to look for a debug netmap kernel, or
> possibly there's something else...
> 
> Thanks,
> 
> -harry
> 
> 
> 
> I can't really say anything without a stack trace or meaningful logs.
> There is a thing that you may do to see if the bug comes out of a bad
> interaction between
> emulated netmap and VALE.
> Instead of attaching the vlan interfaces to VALE you can connect VALE to
> the vlan interface
> through the "bridge" program. In this way nothing changes from the
> functional point of view,
> but you are not attaching anymore the VLAN interface to VALE (and you
> are using an additional process).
> 
> So instead of
> 
>   # vale-ctl vale0:vlan0
> 
> you would have
> 
>   # bridge netmap:vlan0 vale0:vv  # "vv" can be anything

Hello Vincenzo,

thank you very much for that interesting hint.
I prepared a netgraph setup yesterday evening, but I'll try your
suggestion first.  Unfortunately I don't have time to prepare a debug
kernel until next reboot (today evening), but maybe the result of this
config is interesting/meaningful using the same kernel.

Will keep you informed, thanks,

-harry
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/vale periodic deadlock

[Vincenzo Maffione]

2017-11-21 21:48 GMT+01:00 Harry Schmalzbauer :

> Bezüglich Vincenzo Maffione's Nachricht vom 21.11.2017 09:39 (localtime):
> …
> >
> > If this is the case, although you are allowed to do that, I don't think
> > it's a convenient way to use netmap.
> > Since VLAN interfaces like vlan0 do not have (and cannot have) native
> > netmap support, you are falling back to emulated netmap adapters (which
> > are probably buggy on FreeBSD, specially when combined with VALE).
> > Apart from bugs I think that with this setup you can't get decent
> > performance that would justify using netmap rather than the standard
> > kernel bridge and TAP devices.
>
> Hello,
>
> lockup happened earlier than expected.
> This time 'vale-ctl' still reported (-l) the configuration.
> One guest, using if_vtnet(4)-virtio-net#vale2:korso, showed:
> dmz: watchdog timeout on queue 0
> (dmz is the renamed if_vtnet(4))
>
> I could attach tcpdump to the uplink interface and also to all vlan
> children.
> Complete silence everywhere.  So it seems the nic stopped processing
> anything.
>
> Do you think that symptom could be caused by my special vale
> integration, so that bugs in netmap emulation could crash the NIC?
> Or is it unlikely that this is related.
>
> I hadn't prepared a debug kernel for the host, so the machine rebooted
> without again.
> I think I'll have to start with replacing vale first, to narrow down
> possible causes.  Today I was lucky, the lockup happend after business
> hours, but I won't rely on that.
> At least I know if I really need to look for a debug netmap kernel, or
> possibly there's something else...
>
> Thanks,
>
> -harry
>


I can't really say anything without a stack trace or meaningful logs.
There is a thing that you may do to see if the bug comes out of a bad
interaction between
emulated netmap and VALE.
Instead of attaching the vlan interfaces to VALE you can connect VALE to
the vlan interface
through the "bridge" program. In this way nothing changes from the
functional point of view,
but you are not attaching anymore the VLAN interface to VALE (and you are
using an additional process).

So instead of

  # vale-ctl vale0:vlan0

you would have

  # bridge netmap:vlan0 vale0:vv  # "vv" can be anything

Cheers,
  Vincenzo

-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/vale periodic deadlock

[Harry Schmalzbauer]

Bezüglich Vincenzo Maffione's Nachricht vom 21.11.2017 09:39 (localtime):
…
> 
> If this is the case, although you are allowed to do that, I don't think
> it's a convenient way to use netmap.
> Since VLAN interfaces like vlan0 do not have (and cannot have) native
> netmap support, you are falling back to emulated netmap adapters (which
> are probably buggy on FreeBSD, specially when combined with VALE).
> Apart from bugs I think that with this setup you can't get decent
> performance that would justify using netmap rather than the standard
> kernel bridge and TAP devices.

Hello,

lockup happened earlier than expected.
This time 'vale-ctl' still reported (-l) the configuration.
One guest, using if_vtnet(4)-virtio-net#vale2:korso, showed:
dmz: watchdog timeout on queue 0
(dmz is the renamed if_vtnet(4))

I could attach tcpdump to the uplink interface and also to all vlan
children.
Complete silence everywhere.  So it seems the nic stopped processing
anything.

Do you think that symptom could be caused by my special vale
integration, so that bugs in netmap emulation could crash the NIC?
Or is it unlikely that this is related.

I hadn't prepared a debug kernel for the host, so the machine rebooted
without again.
I think I'll have to start with replacing vale first, to narrow down
possible causes.  Today I was lucky, the lockup happend after business
hours, but I won't rely on that.
At least I know if I really need to look for a debug netmap kernel, or
possibly there's something else...

Thanks,

-harry
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/vale periodic deadlock

[Harry Schmalzbauer]

Bezüglich Vincenzo Maffione's Nachricht vom 21.11.2017 14:26 (localtime):
> It may be that your is not a deadlock but some kind of crash. Enabling
> debugging features would probably help (e.g. to get a stack trace).
> Maybe your lockup/crash happened because you did some reconfiguration
> (ring size, number of rings, etc.) while netmap was active and doing so
> you triggered
> some hidden bug.

The host was completely untouched when these lockups occured in late
test phase.  Only guests were configured/utilized.

No previous (short term stress) test had caused any problem in that
path.  It first showed up with real-world (unstressed) tests.

The last-minute change I described was with powered down guests and the
host was rebooted (ppt dev changed in loader.conf).
The host isn't going to be reconfigured in any way.

Let's wait and see if the lockup shows up again (after not limiting NICs
rx/tx descriptors and increasing netmap ring size).

Considering your suspect that emulated netmap code in FreeBSD might be
buggy, and the fact that I'm not able to debug it myself, I guess
switching from vale to netgraph is my best bet.  It's not much effort
and causes almost no downtime.  But it would disallow future ptnetmap
extension...
I'd prefere to stay with vale, although I'm using emulated mode only...
So at first occurance, I'll install the debug kernel and see if that
makes any difference.

Thanks,

-harry
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap/vale periodic deadlock

[Vincenzo Maffione]

It may be that your is not a deadlock but some kind of crash. Enabling
debugging features would probably help (e.g. to get a stack trace).
Maybe your lockup/crash happened because you did some reconfiguration (ring
size, number of rings, etc.) while netmap was active and doing so you
triggered
some hidden bug.

You probably need to enable all these options
https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-options.html



2017-11-21 10:58 GMT+01:00 Harry Schmalzbauer :

> Bezüglich Vincenzo Maffione's Nachricht vom 21.11.2017 09:39 (localtime):
> > Hi,
> >   It's hard to say, specially because it happens after two days of
> > normal use.
> > Can't you enable deadlock debugging features in your kernel?
> > https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/
> kerneldebug-deadlocks.html
> >
> > However, if I understand correctly you have created some VLAN interfaces
> > vlan0, vlan1, vlan2, ... on top of a NIC (say em0). And you have
> > attached each VLAN interface to a vale switch:
> >
> > # vale-ctl -a vale0:vlan0
> > # vale-ctl -a vale1:vlan1
> > # vale-ctl -a vale2:vlan2
> >
> > and each VALE switch is attached to a different set of bhyve guests.
>
> Hello Vincenzo,
>
> thank you very much for your help again!
>
> Your assumption is correct, here's my vale-ctl:
> 603.811416 bdg_ctl [148] bridge:0 port:0 vale1:nic1_dmz
> 603.811428 bdg_ctl [148] bridge:0 port:1 vale1:styx0
> 603.811430 bdg_ctl [148] bridge:0 port:2 vale1:korso
> 603.811432 bdg_ctl [148] bridge:0 port:3 vale1:kallisto
>
> 603.811434 bdg_ctl [148] bridge:1 port:0 vale2:nic1_inop
> 603.811435 bdg_ctl [148] bridge:1 port:1 vale2:styx0
>
> 603.811437 bdg_ctl [148] bridge:2 port:0 vale3:nic1_vnl
> 603.811439 bdg_ctl [148] bridge:2 port:1 vale3:styx0
>
> 603.811441 bdg_ctl [148] bridge:3 port:0 vale4:nic1_egn
> 603.811442 bdg_ctl [148] bridge:3 port:1 vale4:styx0
> 603.811444 bdg_ctl [148] bridge:3 port:2 vale4:preed
> …
>
>
> > If this is the case, although you are allowed to do that, I don't think
> > it's a convenient way to use netmap.
> > Since VLAN interfaces like vlan0 do not have (and cannot have) native
> > netmap support, you are falling back to emulated netmap adapters (which
> > are probably buggy on FreeBSD, specially when combined with VALE).
> > Apart from bugs I think that with this setup you can't get decent
> > performance that would justify using netmap rather than the standard
> > kernel bridge and TAP devices.
>
> I'm aware about the lost netmap-performace-benefit due to emulated
> netmap fallback.
> But there were some resonons why I chose vale(4) instead if_bridge(4):
>
> 1) Inter-Guest-traffic (virtio-net causes lot of LAPIC/IRQ overhead, but
> still less overhead than tap(4)/if_bridge(4))
>
> 2) Future ptnetmap(4) upgrade path (which should save a lot of LAPIC/IRQ
> CPU cycles and unleash huge performace benefits with inter-vm traffic)
>
> 3) Admin-mess and MTU limitation.  Each if_bridge(4) causes a host-stack
> interface, which I don't use and which spams ifconfig(8) output; which
> if_vtnet(4) even doubles.
> Most important disadvantage: if_bridge(4) needs all members to have
> exactly the same MTU.  This has been a problem for me many times over
> the last years in various setups.
>
> So with my current setup the overhead/efficiency of host-external packet
> flow of
> bhyve_virtio-net+dyn_vale_port+vale(4)
> is equal to
> bhyve_virtio-net+if_vtnet(4)+if_bridge(4)
>
> But I have less disadvanteges with vale(4); as long as emulated netmap
> mode doesn't destabilize my setup :-(
>
>
> My second choice was ng_bridge(4).  Which I made great experiences in my
> router-vm, running on that host in question (and in turn uses virtio-net
> interfaces attached to the individual vale(4) switches on the host).
> [ Even more impressive:  pf(4) runs in a VIMAGE jail in that guest,
> utilizing those vale(4) interfaces.  Reason for that complicated setup:
>  Closest hardware abstraction possible.  The setup (guest) should be
> easily migratable to real hardware ].
>
>
> > The right way to do it imho would be to write your own (userspace)
> > netmap application that forwards packets between your bhyve guests and
> > the NIC, prepending/stripping VLAN headers according to configuration
> > (e.g. guest A is configured to be on VLAN 100, guest B on VLAN 200), etc.
> > I think this would be a very interesting netmap application in general,
> > and more importantly you would get the performance that you can't get
> > with your setup.
>
> I agree that having a userland application which, like you described,
> utilizes netmap to enable minimalistic SDN features, would be a great
> solution.  But I would need really a lot of time, since my C skills are
> lousy, and I really don't have any time, not even one more day.
>

I see. But just FYI, there isn't that much to implement :)

Cheers,
  Vincenzo


>
>
> I'll see if I can get any useful information with 

Re: netmap/vale periodic deadlock

[Harry Schmalzbauer]

Bezüglich Vincenzo Maffione's Nachricht vom 21.11.2017 09:39 (localtime):
> Hi,
>   It's hard to say, specially because it happens after two days of
> normal use.
> Can't you enable deadlock debugging features in your kernel?
> https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html
> 
> However, if I understand correctly you have created some VLAN interfaces
> vlan0, vlan1, vlan2, ... on top of a NIC (say em0). And you have
> attached each VLAN interface to a vale switch:
> 
> # vale-ctl -a vale0:vlan0
> # vale-ctl -a vale1:vlan1
> # vale-ctl -a vale2:vlan2
> 
> and each VALE switch is attached to a different set of bhyve guests.

Hello Vincenzo,

thank you very much for your help again!

Your assumption is correct, here's my vale-ctl:
603.811416 bdg_ctl [148] bridge:0 port:0 vale1:nic1_dmz
603.811428 bdg_ctl [148] bridge:0 port:1 vale1:styx0
603.811430 bdg_ctl [148] bridge:0 port:2 vale1:korso
603.811432 bdg_ctl [148] bridge:0 port:3 vale1:kallisto

603.811434 bdg_ctl [148] bridge:1 port:0 vale2:nic1_inop
603.811435 bdg_ctl [148] bridge:1 port:1 vale2:styx0

603.811437 bdg_ctl [148] bridge:2 port:0 vale3:nic1_vnl
603.811439 bdg_ctl [148] bridge:2 port:1 vale3:styx0

603.811441 bdg_ctl [148] bridge:3 port:0 vale4:nic1_egn
603.811442 bdg_ctl [148] bridge:3 port:1 vale4:styx0
603.811444 bdg_ctl [148] bridge:3 port:2 vale4:preed
…


> If this is the case, although you are allowed to do that, I don't think
> it's a convenient way to use netmap.
> Since VLAN interfaces like vlan0 do not have (and cannot have) native
> netmap support, you are falling back to emulated netmap adapters (which
> are probably buggy on FreeBSD, specially when combined with VALE).
> Apart from bugs I think that with this setup you can't get decent
> performance that would justify using netmap rather than the standard
> kernel bridge and TAP devices.

I'm aware about the lost netmap-performace-benefit due to emulated
netmap fallback.
But there were some resonons why I chose vale(4) instead if_bridge(4):

1) Inter-Guest-traffic (virtio-net causes lot of LAPIC/IRQ overhead, but
still less overhead than tap(4)/if_bridge(4))

2) Future ptnetmap(4) upgrade path (which should save a lot of LAPIC/IRQ
CPU cycles and unleash huge performace benefits with inter-vm traffic)

3) Admin-mess and MTU limitation.  Each if_bridge(4) causes a host-stack
interface, which I don't use and which spams ifconfig(8) output; which
if_vtnet(4) even doubles.
Most important disadvantage: if_bridge(4) needs all members to have
exactly the same MTU.  This has been a problem for me many times over
the last years in various setups.

So with my current setup the overhead/efficiency of host-external packet
flow of
bhyve_virtio-net+dyn_vale_port+vale(4)
is equal to
bhyve_virtio-net+if_vtnet(4)+if_bridge(4)

But I have less disadvanteges with vale(4); as long as emulated netmap
mode doesn't destabilize my setup :-(


My second choice was ng_bridge(4).  Which I made great experiences in my
router-vm, running on that host in question (and in turn uses virtio-net
interfaces attached to the individual vale(4) switches on the host).
[ Even more impressive:  pf(4) runs in a VIMAGE jail in that guest,
utilizing those vale(4) interfaces.  Reason for that complicated setup:
 Closest hardware abstraction possible.  The setup (guest) should be
easily migratable to real hardware ].


> The right way to do it imho would be to write your own (userspace)
> netmap application that forwards packets between your bhyve guests and
> the NIC, prepending/stripping VLAN headers according to configuration
> (e.g. guest A is configured to be on VLAN 100, guest B on VLAN 200), etc.
> I think this would be a very interesting netmap application in general,
> and more importantly you would get the performance that you can't get
> with your setup.

I agree that having a userland application which, like you described,
utilizes netmap to enable minimalistic SDN features, would be a great
solution.  But I would need really a lot of time, since my C skills are
lousy, and I really don't have any time, not even one more day.


I'll see if I can get any useful information with the kernel deadlock
debuging feature you suggested
(https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html),
as soon as the problem shows up again.
Since I forgot to add all production-RAM, I had to shutdown yesterday,
so the lockup counter was reset ;-)
Another last-minute change was with netmap ring size: I changed the
vale-uplink interface.  The one I used for passthrough had 2 queues
(with EM_MULTIQUEUE support) and the one for the vale uplink onyl one,
and during evaluation phase I reduced rx/tx descriptors to make netmap's
default ring size working.
Now I use the 2-queue NIC with vale uplink and increased ring size to
81920 while leaving the hardware default of 4096 rx/tx desriptors.

But my wording wasn't technically correct I 

Re: netmap/vale periodic deadlock

[Vincenzo Maffione]

Hi,
  It's hard to say, specially because it happens after two days of normal
use.
Can't you enable deadlock debugging features in your kernel?
https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html

However, if I understand correctly you have created some VLAN interfaces
vlan0, vlan1, vlan2, ... on top of a NIC (say em0). And you have attached
each VLAN interface to a vale switch:

# vale-ctl -a vale0:vlan0
# vale-ctl -a vale1:vlan1
# vale-ctl -a vale2:vlan2

and each VALE switch is attached to a different set of bhyve guests.

If this is the case, although you are allowed to do that, I don't think
it's a convenient way to use netmap.
Since VLAN interfaces like vlan0 do not have (and cannot have) native
netmap support, you are falling back to emulated netmap adapters (which are
probably buggy on FreeBSD, specially when combined with VALE).
Apart from bugs I think that with this setup you can't get decent
performance that would justify using netmap rather than the standard kernel
bridge and TAP devices.

The right way to do it imho would be to write your own (userspace) netmap
application that forwards packets between your bhyve guests and the NIC,
prepending/stripping VLAN headers according to configuration (e.g. guest A
is configured to be on VLAN 100, guest B on VLAN 200), etc.
I think this would be a very interesting netmap application in general, and
more importantly you would get the performance that you can't get with your
setup.

Cheers,
  Vincenzo

2017-11-17 17:56 GMT+01:00 Harry Schmalzbauer :

>  Hello,
>
> sorry for annoying with another question/problem.
>
> I'm using netmap's vale (on stable/11) for bhyve(8) virtio-net backed SDN.
>
> The guests – unfortunately in production already – quit network services
> (resp. are not able to transceive any packets anymore) after about 2
> days; repeatedly and most likely not load related, since there is no
> significant load.
> Each guest is running fine, the host also runs without any other
> problem, no network problem elsewhere (different NICs; I use one
> dedicated NIC with vlan(4) children, each child connected to one vale
> switch).
>
> At some point, the complete netmap subsystem seems to deadlock:
> 'vale-ctl' hangs uninteruptable.
> Trying to attach a tcpdump to a vale switch also hands uninteruptable.
> Stoping (shuting down from inside) bhyve guests works up to the point
> where the vale port should be destroyed.
> I could continue the list of symptoms, but that doesn't help in any way
> I guess.
>
> My question is, where can I start finding out what happens with the
> netmap subsystem?
>
> There were no kernel messages right before or during the deadlock!
>
> The only userland tool I'm familar with (vale-ctl) isn't usable at all
> in that situation.
> Any hints what to try?
>
>
> Here's a excerpt of processes running when the netmap-lockuped host has
> all guests shut down, just before I rebooted.
> Snipped alot, the interesing ones are thos in state "netmap_g":
> …
> 0 14213 1 0 20 0 5864 0 wait IW 3 0:00,00 (sh)
> 0 14214 14213 0 -92 0 5358120 3586232 nm_kn_lo TC 3 148:02,02 bhyve:
> kallisto (bhyve)
> 0 14976 2522 0 20 0 6976 0 wait IW 3 0:00,00 su
> 0 14981 14976 0 20 0 8256 0 pause IW 3 0:00,00 _su (csh)
> 0 61615 14981 0 20 0 5864 0 wait IW 3 0:00,00 (sh)
> 0 61616 61615 0 52 0 2180648 1973252 netmap_g DEC 3 286:11,91 bhyve:
> preed (bhyve)
> 0 62845 14981 0 20 0 11624 3328 bdg lock L+ 3 0:00,01 tcpdump -n -e -s
> 150 -i vale1:test
> …
> 0 1390 1388 0 -92 0 2330024 767756 nm_kn_lo TC v0- 94:01,90 bhyve: styx0
> (bhyve)
> 0 1401 1 0 52 0 5784 0 wait IW v0- 0:00,00 (sh)
> 0 1403 1401 0 20 0 368328 43444 - TC v0- 3:35,66 bhyve: korso (bhyve)
> …
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: [netmap] when does a packet in the netmap ring send out exactly

[Vincenzo Maffione]

Hi,
  Netmap version (tag v11.3 vs master) should be unrelated.
Are you using ixgbe? In this case it may depend on the ixgbe driver version
that netmap is using for patching.
On github master you can change the version by modifying the ixgbe line in
LINUX/default-config.mak.in_.
Valid versions are for instance 4.5.4 or 5.1.3.

Cheers,
  Vincenzo

2017-11-21 4:43 GMT+01:00 Xiaoye Sun :

> Hi Luigi,
>
> Thanks!
> I was using the most recent netmap on Github and I believe the tail
> pointer only moves forward when there are less than half of the total slots
> available in the netmap ring.
> Then I switch to the version of v11.3
> , it behaves as what you
> described.
>
> Linux Kernel: 3.16.0-4-amd64
>
> Best,
> Xiaoye
>
>
> On Mon, Nov 20, 2017 at 6:11 PM, Luigi Rizzo  wrote:
>
>> Hi,
>> I think if you call the TXSYNC ioctl without advancing the head
>> pointer, then the tail is advanced
>> as much as possible.
>>
>> Cheers
>> luigi
>>
>> On Mon, Nov 20, 2017 at 3:35 PM, Xiaoye Sun  wrote:
>> > Hi,
>> >
>> > I found that the tail pointer only moves when the ring has less than
>> half
>> > of the slots available. This prevents me from knowing the accurate time
>> > when the packet in a slot is processed. Is there a way to move the tail
>> > pointer as long as the packet in the slot is processed? Is this a
>> > configurable feature?
>> >
>> > Best,
>> > Xiaoye
>> >
>> > On Fri, Oct 27, 2017 at 11:52 AM, Vincenzo Maffione <
>> v.maffi...@gmail.com>
>> > wrote:
>> >
>> >> Hi,
>> >>   This is actually a limitation of the netmap API: ring->tail is
>> exposed
>> >> to the user so that it knows it can use the slots in the range
>> >> "[ring->head..ring->tail[" for new transmissions (note that head is
>> >> included, tail excluded, to prevent wraparound). However, there is no
>> >> explicit indication of "up to what slots packets were transmitted".
>> >> For hw NICs, however, ring->tail is an indication of where transmission
>> >> was completed.
>> >> Example:
>> >> 1) at the beginning ring->tail = ring->head = ring->cur = 0
>> >> 2) then your program moves head/cur forward: head = cur = 10
>> >> 3) you call TXSYNC, to submit the packets to the NIC.
>> >> 4) after the TXSYNC call, is very likely that tail is still 0, i.e.
>> >> because no transmission has been completed by the NIC (and no interrupt
>> >> generated).
>> >> 5) say after 20 us you issue another TXSYNC,  and in the meanwhile 6
>> >> packets had completed. In this case after TXSYNC you will find tail==5,
>> >> meaning that packets in the slots 0,1,2,3,4 and 5 have been completed.
>> Note
>> >> that also the slot pointed by tail has been completed.
>> >>
>> >> But you are right that there is no way to receive completion
>> notification
>> >> if the queue is not full. You must use TXSYNC to check (by sleeping or
>> busy
>> >> wait) when tail moves forward.
>> >>
>> >> Cheers,
>> >>   Vincenzo
>> >>
>> >>
>> >> 2017-10-27 3:06 GMT+02:00 Xiaoye Sun :
>> >>
>> >>> Hi
>> >>>
>> >>> I write a netmap program that sends packets to the network. my program
>> >>> uses one netmap ring and fills the ring slots with packets.
>> >>> My program needs to do something (action A) after a particular packet
>> >>> (packet P) in the ring slot is sent to the network. so the program
>> tracks
>> >>> the position of the tail point and checks if the tail point has moved
>> >>> across the slot I used to put that packet P.
>> >>> However, I found that the tail pointer may not move forward even
>> seconds
>> >>> after the receiver side got packet P.
>> >>> Sometimes the tail pointer never moves forward until the TX ring is
>> full.
>> >>> I try ioctl(NIOCTXSYNC), however, it cannot 100% solve the problem.
>> >>>
>> >>> My question is that is there a way to make the TX ring empty as early
>> as
>> >>> possible so that I can know when my packet is sent out. or is there
>> >>> another
>> >>> way to know when the packet in the slot is sent to the network/NIC
>> >>> physical
>> >>> queue?
>> >>>
>> >>> I am using Linux 3.16.0-4-amd64.
>> >>>
>> >>> Thanks!
>> >>>
>> >>> Best,
>> >>> Xiaoye
>> >>> ___
>> >>> freebsd-net@freebsd.org mailing list
>> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> >>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org
>> "
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Vincenzo Maffione
>> >>
>> > ___
>> > freebsd-net@freebsd.org mailing list
>> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>
>>
>>
>> --
>> -+---
>>  Prof. Luigi RIZZO, ri...@iet.unipi.it  . Dip. di Ing. dell'Informazione
>>  http://www.iet.unipi.it/~luigi/. Universita` di Pisa
>>  TEL  

Re: [netmap] when does a packet in the netmap ring send out exactly

[Xiaoye Sun]

Hi Luigi,

Thanks!
I was using the most recent netmap on Github and I believe the tail pointer
only moves forward when there are less than half of the total slots
available in the netmap ring.
Then I switch to the version of v11.3
, it behaves as what you
described.

Linux Kernel: 3.16.0-4-amd64

Best,
Xiaoye


On Mon, Nov 20, 2017 at 6:11 PM, Luigi Rizzo  wrote:

> Hi,
> I think if you call the TXSYNC ioctl without advancing the head
> pointer, then the tail is advanced
> as much as possible.
>
> Cheers
> luigi
>
> On Mon, Nov 20, 2017 at 3:35 PM, Xiaoye Sun  wrote:
> > Hi,
> >
> > I found that the tail pointer only moves when the ring has less than half
> > of the slots available. This prevents me from knowing the accurate time
> > when the packet in a slot is processed. Is there a way to move the tail
> > pointer as long as the packet in the slot is processed? Is this a
> > configurable feature?
> >
> > Best,
> > Xiaoye
> >
> > On Fri, Oct 27, 2017 at 11:52 AM, Vincenzo Maffione <
> v.maffi...@gmail.com>
> > wrote:
> >
> >> Hi,
> >>   This is actually a limitation of the netmap API: ring->tail is exposed
> >> to the user so that it knows it can use the slots in the range
> >> "[ring->head..ring->tail[" for new transmissions (note that head is
> >> included, tail excluded, to prevent wraparound). However, there is no
> >> explicit indication of "up to what slots packets were transmitted".
> >> For hw NICs, however, ring->tail is an indication of where transmission
> >> was completed.
> >> Example:
> >> 1) at the beginning ring->tail = ring->head = ring->cur = 0
> >> 2) then your program moves head/cur forward: head = cur = 10
> >> 3) you call TXSYNC, to submit the packets to the NIC.
> >> 4) after the TXSYNC call, is very likely that tail is still 0, i.e.
> >> because no transmission has been completed by the NIC (and no interrupt
> >> generated).
> >> 5) say after 20 us you issue another TXSYNC,  and in the meanwhile 6
> >> packets had completed. In this case after TXSYNC you will find tail==5,
> >> meaning that packets in the slots 0,1,2,3,4 and 5 have been completed.
> Note
> >> that also the slot pointed by tail has been completed.
> >>
> >> But you are right that there is no way to receive completion
> notification
> >> if the queue is not full. You must use TXSYNC to check (by sleeping or
> busy
> >> wait) when tail moves forward.
> >>
> >> Cheers,
> >>   Vincenzo
> >>
> >>
> >> 2017-10-27 3:06 GMT+02:00 Xiaoye Sun :
> >>
> >>> Hi
> >>>
> >>> I write a netmap program that sends packets to the network. my program
> >>> uses one netmap ring and fills the ring slots with packets.
> >>> My program needs to do something (action A) after a particular packet
> >>> (packet P) in the ring slot is sent to the network. so the program
> tracks
> >>> the position of the tail point and checks if the tail point has moved
> >>> across the slot I used to put that packet P.
> >>> However, I found that the tail pointer may not move forward even
> seconds
> >>> after the receiver side got packet P.
> >>> Sometimes the tail pointer never moves forward until the TX ring is
> full.
> >>> I try ioctl(NIOCTXSYNC), however, it cannot 100% solve the problem.
> >>>
> >>> My question is that is there a way to make the TX ring empty as early
> as
> >>> possible so that I can know when my packet is sent out. or is there
> >>> another
> >>> way to know when the packet in the slot is sent to the network/NIC
> >>> physical
> >>> queue?
> >>>
> >>> I am using Linux 3.16.0-4-amd64.
> >>>
> >>> Thanks!
> >>>
> >>> Best,
> >>> Xiaoye
> >>> ___
> >>> freebsd-net@freebsd.org mailing list
> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> >>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
> >>>
> >>
> >>
> >>
> >> --
> >> Vincenzo Maffione
> >>
> > ___
> > freebsd-net@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
>
>
> --
> -+---
>  Prof. Luigi RIZZO, ri...@iet.unipi.it  . Dip. di Ing. dell'Informazione
>  http://www.iet.unipi.it/~luigi/. Universita` di Pisa
>  TEL  +39-050-2217533   . via Diotisalvi 2
>  Mobile   +39-338-6809875   . 56122 PISA (Italy)
> -+---
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: [netmap] when does a packet in the netmap ring send out exactly

[Luigi Rizzo]

Hi,
I think if you call the TXSYNC ioctl without advancing the head
pointer, then the tail is advanced
as much as possible.

Cheers
luigi

On Mon, Nov 20, 2017 at 3:35 PM, Xiaoye Sun  wrote:
> Hi,
>
> I found that the tail pointer only moves when the ring has less than half
> of the slots available. This prevents me from knowing the accurate time
> when the packet in a slot is processed. Is there a way to move the tail
> pointer as long as the packet in the slot is processed? Is this a
> configurable feature?
>
> Best,
> Xiaoye
>
> On Fri, Oct 27, 2017 at 11:52 AM, Vincenzo Maffione 
> wrote:
>
>> Hi,
>>   This is actually a limitation of the netmap API: ring->tail is exposed
>> to the user so that it knows it can use the slots in the range
>> "[ring->head..ring->tail[" for new transmissions (note that head is
>> included, tail excluded, to prevent wraparound). However, there is no
>> explicit indication of "up to what slots packets were transmitted".
>> For hw NICs, however, ring->tail is an indication of where transmission
>> was completed.
>> Example:
>> 1) at the beginning ring->tail = ring->head = ring->cur = 0
>> 2) then your program moves head/cur forward: head = cur = 10
>> 3) you call TXSYNC, to submit the packets to the NIC.
>> 4) after the TXSYNC call, is very likely that tail is still 0, i.e.
>> because no transmission has been completed by the NIC (and no interrupt
>> generated).
>> 5) say after 20 us you issue another TXSYNC,  and in the meanwhile 6
>> packets had completed. In this case after TXSYNC you will find tail==5,
>> meaning that packets in the slots 0,1,2,3,4 and 5 have been completed. Note
>> that also the slot pointed by tail has been completed.
>>
>> But you are right that there is no way to receive completion notification
>> if the queue is not full. You must use TXSYNC to check (by sleeping or busy
>> wait) when tail moves forward.
>>
>> Cheers,
>>   Vincenzo
>>
>>
>> 2017-10-27 3:06 GMT+02:00 Xiaoye Sun :
>>
>>> Hi
>>>
>>> I write a netmap program that sends packets to the network. my program
>>> uses one netmap ring and fills the ring slots with packets.
>>> My program needs to do something (action A) after a particular packet
>>> (packet P) in the ring slot is sent to the network. so the program tracks
>>> the position of the tail point and checks if the tail point has moved
>>> across the slot I used to put that packet P.
>>> However, I found that the tail pointer may not move forward even seconds
>>> after the receiver side got packet P.
>>> Sometimes the tail pointer never moves forward until the TX ring is full.
>>> I try ioctl(NIOCTXSYNC), however, it cannot 100% solve the problem.
>>>
>>> My question is that is there a way to make the TX ring empty as early as
>>> possible so that I can know when my packet is sent out. or is there
>>> another
>>> way to know when the packet in the slot is sent to the network/NIC
>>> physical
>>> queue?
>>>
>>> I am using Linux 3.16.0-4-amd64.
>>>
>>> Thanks!
>>>
>>> Best,
>>> Xiaoye
>>> ___
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>>
>>
>>
>>
>> --
>> Vincenzo Maffione
>>
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"



-- 
-+---
 Prof. Luigi RIZZO, ri...@iet.unipi.it  . Dip. di Ing. dell'Informazione
 http://www.iet.unipi.it/~luigi/. Universita` di Pisa
 TEL  +39-050-2217533   . via Diotisalvi 2
 Mobile   +39-338-6809875   . 56122 PISA (Italy)
-+---
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: [netmap] when does a packet in the netmap ring send out exactly

[Xiaoye Sun]

Hi,

I found that the tail pointer only moves when the ring has less than half
of the slots available. This prevents me from knowing the accurate time
when the packet in a slot is processed. Is there a way to move the tail
pointer as long as the packet in the slot is processed? Is this a
configurable feature?

Best,
Xiaoye

On Fri, Oct 27, 2017 at 11:52 AM, Vincenzo Maffione 
wrote:

> Hi,
>   This is actually a limitation of the netmap API: ring->tail is exposed
> to the user so that it knows it can use the slots in the range
> "[ring->head..ring->tail[" for new transmissions (note that head is
> included, tail excluded, to prevent wraparound). However, there is no
> explicit indication of "up to what slots packets were transmitted".
> For hw NICs, however, ring->tail is an indication of where transmission
> was completed.
> Example:
> 1) at the beginning ring->tail = ring->head = ring->cur = 0
> 2) then your program moves head/cur forward: head = cur = 10
> 3) you call TXSYNC, to submit the packets to the NIC.
> 4) after the TXSYNC call, is very likely that tail is still 0, i.e.
> because no transmission has been completed by the NIC (and no interrupt
> generated).
> 5) say after 20 us you issue another TXSYNC,  and in the meanwhile 6
> packets had completed. In this case after TXSYNC you will find tail==5,
> meaning that packets in the slots 0,1,2,3,4 and 5 have been completed. Note
> that also the slot pointed by tail has been completed.
>
> But you are right that there is no way to receive completion notification
> if the queue is not full. You must use TXSYNC to check (by sleeping or busy
> wait) when tail moves forward.
>
> Cheers,
>   Vincenzo
>
>
> 2017-10-27 3:06 GMT+02:00 Xiaoye Sun :
>
>> Hi
>>
>> I write a netmap program that sends packets to the network. my program
>> uses one netmap ring and fills the ring slots with packets.
>> My program needs to do something (action A) after a particular packet
>> (packet P) in the ring slot is sent to the network. so the program tracks
>> the position of the tail point and checks if the tail point has moved
>> across the slot I used to put that packet P.
>> However, I found that the tail pointer may not move forward even seconds
>> after the receiver side got packet P.
>> Sometimes the tail pointer never moves forward until the TX ring is full.
>> I try ioctl(NIOCTXSYNC), however, it cannot 100% solve the problem.
>>
>> My question is that is there a way to make the TX ring empty as early as
>> possible so that I can know when my packet is sent out. or is there
>> another
>> way to know when the packet in the slot is sent to the network/NIC
>> physical
>> queue?
>>
>> I am using Linux 3.16.0-4-amd64.
>>
>> Thanks!
>>
>> Best,
>> Xiaoye
>> ___
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>
>
>
>
> --
> Vincenzo Maffione
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap scatter/gather?

[K. Macy]

On Sat, Nov 11, 2017 at 8:30 PM, K. Macy  wrote:
> On Tue, Nov 7, 2017 at 9:32 AM, Vincenzo Maffione  
> wrote:
>> Hi,
>>   In general netmap adapters (i.e. netmap ports) may support NS_MOREFRAG.
>> But in practice this is mainly supported on VALE ports.
>> So if you don't want to add the missing support by yourself you can simply
>> change the netmap buffer size by tuning the sysctl dev.netmap.buf_size, and
>> increase it to 9600.
>>
>
> When doing vxlan that allows me to avoid copies on encap, but I don't
> see any mechanism to avoid a (second) copy on decap where what I'd
> like to do is to indicate to VALE that the packet starts at a certain
> offset. Am I missing something?

It looks like if you'd just change NETMAP_BUF to take an offset as
well it would mostly just work if I could pass on offset in
netmap_slot:

idx = ring->slot[i].buf_idx;
d->hdr.slot = >slot[i];
d->hdr.buf = (u_char *)NETMAP_BUF(ring, idx);

->
idx = ring->slot[i].buf_idx;
off = ring->slot[i].buf_off; /* new */
d->hdr.slot = >slot[i];
d->hdr.buf = (u_char *)NETMAP_BUF(ring, idx, off);


>
>
> Thanks.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap scatter/gather?

[K. Macy]

On Tue, Nov 7, 2017 at 9:32 AM, Vincenzo Maffione  wrote:
> Hi,
>   In general netmap adapters (i.e. netmap ports) may support NS_MOREFRAG.
> But in practice this is mainly supported on VALE ports.
> So if you don't want to add the missing support by yourself you can simply
> change the netmap buffer size by tuning the sysctl dev.netmap.buf_size, and
> increase it to 9600.
>

When doing vxlan that allows me to avoid copies on encap, but I don't
see any mechanism to avoid a (second) copy on decap where what I'd
like to do is to indicate to VALE that the packet starts at a certain
offset. Am I missing something?


Thanks.
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap scatter/gather?

[Vincenzo Maffione]

I see, but it seems there is a bug anyway. Feel free to open a bug report
here https://github.com/luigirizzo/netmap/issues if you wish.

2017-11-07 21:13 GMT+01:00 Joe Buehler :

> Decreasing buf_num to 32768 eliminated the allocation failure.
>
> Joe Buehler
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap scatter/gather?

[Joe Buehler]

Decreasing buf_num to 32768 eliminated the allocation failure.

Joe Buehler
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap scatter/gather?

[Joe Buehler]

Vincenzo Maffione wrote:

>   In general netmap adapters (i.e. netmap ports) may support
> NS_MOREFRAG. But in practice this is mainly supported on VALE ports.
> So if you don't want to add the missing support by yourself you can
> simply change the netmap buffer size by tuning the sysctl
> dev.netmap.buf_size, and increase it to 9600.

I added parameter buf_size=9600 to the LINUX netmap module.  The TX path
works properly, it sends 9600 byte frames, but my app receives no RX
frames from the loopback.

I presume this is why:

[  856.820521] 618.970159 [ 716] virtio_netmap_config  virtio config
txq=1, txd=256 rxq=1, rxd=256
[  857.972334] daemon_exe: page allocation failure: order:7, mode:0x2088020
[  857.972345] CPU: 1 PID: 2289 Comm: daemon_exe Tainted: G   OE
  4.4.86rt-rt99 #0
[  857.972346] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[  857.972350]  0286 36d1c578027e2926 88003d3df938
813ecf3e
[  857.972352]  02088020  88003d3df9c8
811a250a
[  857.972352]  88003ffde078  88003ffdc020
88003d3df990
[  857.972353] Call Trace:
[  857.972384]  [] dump_stack+0x63/0x85
[  857.972395]  [] warn_alloc_failed+0xfa/0x150
[  857.972397]  [] __alloc_pages_nodemask+0x352/0xb80
[  857.972409]  [] alloc_pages_current+0x8d/0x110
[  857.972428]  [] netmap_mem2_finalize+0x142/0x590
[netmap]
[  857.972443]  [] ? virtio_netmap_config+0xcc/0x100
[virtio_net]
[  857.972446]  [] netmap_mem_finalize+0x83/0x2c0 [netmap]
[  857.972449]  [] netmap_do_regif+0x89/0x2e0 [netmap]
[  857.972452]  [] netmap_ioctl+0x5fb/0xa50 [netmap]
[  857.972462]  [] ? migrate_enable+0x7a/0x150
[  857.972473]  [] ? rt_spin_unlock+0x27/0x40
[  857.972479]  [] ? lockref_put_or_lock+0x25/0x30
[  857.972488]  [] ? mntput_no_expire+0x2c/0x1b0
[  857.972489]  [] ? mntput+0x24/0x40
[  857.972493]  [] ? terminate_walk+0x6b/0xe0
[  857.972496]  [] linux_netmap_ioctl+0xa9/0x120 [netmap]
[  857.972497]  [] ? do_filp_open+0xa5/0x100
[  857.972499]  [] ? migrate_enable+0x7a/0x150
[  857.972500]  [] do_vfs_ioctl+0x298/0x490
[  857.972502]  [] ? __fget+0x77/0xb0
[  857.972503]  [] SyS_ioctl+0x79/0x90
[  857.972504]  [] entry_SYSCALL_64_fastpath+0x16/0x71
[  857.972511] Mem-Info:
[  857.972522] active_anon:16975 inactive_anon:684 isolated_anon:0
active_file:13080 inactive_file:24152 isolated_file:0
unevictable:1466 dirty:3 writeback:0 unstable:0
slab_reclaimable:6202 slab_unreclaimable:3621
mapped:13590 shmem:794 pagetables:748 bounce:0
free:1403 free_pcp:274 free_cma:0
[  857.972528] Node 0 DMA free:3776kB min:64kB low:80kB high:96kB
active_anon:1992kB inactive_anon:64kB active_file:1164kB
inactive_file:2408kB unevictable:176kB isolated(anon):0kB
isolated(file):0kB present:15992kB managed:15908kB mlocked:176kB
dirty:0kB writeback:0kB mapped:1460kB shmem:76kB slab_reclaimable:576kB
slab_unreclaimable:388kB kernel_stack:32kB pagetables:76kB unstable:0kB
bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB
pages_scanned:0 all_unreclaimable? no
[  857.972534] lowmem_reserve[]: 0 926 926 926
[  857.972536] Node 0 DMA32 free:1836kB min:3860kB low:4824kB
high:5788kB active_anon:65908kB inactive_anon:2672kB active_file:51156kB
inactive_file:94200kB unevictable:5688kB isolated(anon):0kB
isolated(file):0kB present:1032060kB managed:998044kB mlocked:5688kB
dirty:12kB writeback:0kB mapped:52900kB shmem:3100kB
slab_reclaimable:24232kB slab_unreclaimable:14096kB kernel_stack:3072kB
pagetables:2916kB unstable:0kB bounce:0kB free_pcp:1096kB
local_pcp:228kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
[  857.972548] lowmem_reserve[]: 0 0 0 0
[  857.972550] Node 0 DMA: 1*4kB (M) 3*8kB (UME) 6*16kB (UE) 4*32kB (UE)
1*64kB (U) 3*128kB (UME) 2*256kB (ME) 1*512kB (U) 2*1024kB (UE) 0*2048kB
0*4096kB = 3772kB
[  857.972560] Node 0 DMA32: 1*4kB (U) 1*8kB (U) 6*16kB (UME) 2*32kB
(EH) 2*64kB (UE) 2*128kB (UH) 1*256kB (H) 2*512kB (H) 0*1024kB 0*2048kB
0*4096kB = 1836kB
[  857.972570] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[  857.972571] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[  857.972572] 39070 total pagecache pages
[  857.972578] 0 pages in swap cache
[  857.972580] Swap cache stats: add 0, delete 0, find 0/0
[  857.972581] Free swap  = 0kB
[  857.972582] Total swap = 0kB
[  857.972582] 262013 pages RAM
[  857.972583] 0 pages HighMem/MovableOnly
[  857.972584] 8525 pages reserved
[  857.972584] 0 pages cma reserved
[  857.972585] 0 pages hwpoisoned
[  857.972591] 620.122243 [1316] netmap_finalize_obj_allocator Unable to
create cluster at 44352 for 'netmap_buf' allocator
[  857.981963] 620.131618 [ 202] virtio_netmap_clean_used_rings got 1
used bufs on queue tx-0
[  857.982582] 620.132238 [ 215] virtio_netmap_clean_used_rings got 0
used bufs on 

Re: netmap scatter/gather?

[Vincenzo Maffione]

Hi,
  In general netmap adapters (i.e. netmap ports) may support NS_MOREFRAG.
But in practice this is mainly supported on VALE ports.
So if you don't want to add the missing support by yourself you can simply
change the netmap buffer size by tuning the sysctl dev.netmap.buf_size, and
increase it to 9600.

Cheers,
  Vincenzo

2017-11-07 18:28 GMT+01:00 Joseph H. Buehler :

> Does NS_MOREFRAG work when using netmap with network adaptors (e.g.
> virtio_net)?
>
> I need to send and receive large frames -- 9600 bytes -- but the netmap
> buffer size is only 2048.
>
> Joe Buehler
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: [netmap] when does a packet in the netmap ring send out exactly

[Vincenzo Maffione]

Hi,
  This is actually a limitation of the netmap API: ring->tail is exposed to
the user so that it knows it can use the slots in the range
"[ring->head..ring->tail[" for new transmissions (note that head is
included, tail excluded, to prevent wraparound). However, there is no
explicit indication of "up to what slots packets were transmitted".
For hw NICs, however, ring->tail is an indication of where transmission was
completed.
Example:
1) at the beginning ring->tail = ring->head = ring->cur = 0
2) then your program moves head/cur forward: head = cur = 10
3) you call TXSYNC, to submit the packets to the NIC.
4) after the TXSYNC call, is very likely that tail is still 0, i.e. because
no transmission has been completed by the NIC (and no interrupt generated).
5) say after 20 us you issue another TXSYNC,  and in the meanwhile 6
packets had completed. In this case after TXSYNC you will find tail==5,
meaning that packets in the slots 0,1,2,3,4 and 5 have been completed. Note
that also the slot pointed by tail has been completed.

But you are right that there is no way to receive completion notification
if the queue is not full. You must use TXSYNC to check (by sleeping or busy
wait) when tail moves forward.

Cheers,
  Vincenzo


2017-10-27 3:06 GMT+02:00 Xiaoye Sun :

> Hi
>
> I write a netmap program that sends packets to the network. my program
> uses one netmap ring and fills the ring slots with packets.
> My program needs to do something (action A) after a particular packet
> (packet P) in the ring slot is sent to the network. so the program tracks
> the position of the tail point and checks if the tail point has moved
> across the slot I used to put that packet P.
> However, I found that the tail pointer may not move forward even seconds
> after the receiver side got packet P.
> Sometimes the tail pointer never moves forward until the TX ring is full.
> I try ioctl(NIOCTXSYNC), however, it cannot 100% solve the problem.
>
> My question is that is there a way to make the TX ring empty as early as
> possible so that I can know when my packet is sent out. or is there another
> way to know when the packet in the slot is sent to the network/NIC physical
> queue?
>
> I am using Linux 3.16.0-4-amd64.
>
> Thanks!
>
> Best,
> Xiaoye
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap with bonded interfaces

[Vincenzo Maffione]

Bypass what?

2017-06-28 22:59 GMT-07:00 Paras Jha :

> It's possible to bypass this by unloading and reloading the patched
> network driver
>
> On Thu, Jun 29, 2017 at 12:39 AM, Vincenzo Maffione 
> wrote:
>
>> Hi,
>>   It is an expected behaviour that you cannot open eth4 and eth5 if they
>> are bond, as the device are being used by the lagg pseudo-driver.
>> Since this driver does not have netmap support for the moment being,
>> there is no way you can get the native mode performance if you use lagg.
>>
>> If you just need some failover in your application, you could just
>> implement a simple failover mechanism in your application (e.g. the
>> application opens both netmap:eth4 and netmap:eth5, and decides which one
>> to use for transmission depending on which one is up...).
>>
>> Cheers,
>>   Vincenzo
>>
>> 2017-06-29 4:16 GMT+02:00 Paras Jha :
>>
>>> Hi all,
>>>
>>> I have a bonded interface bond0 which enslaves eth4 and eth5. When trying
>>> to open the devices eth4 or eth5 via netmap, I get a "device in use"
>>> error.
>>> Opening the bond0 interface directly in netmap works, however it is in
>>> emulated mode (as expected of a pseudointerface)
>>>
>>> What is the idiomatic way to proceed in such a situation, without
>>> compromising on speed?
>>>
>>> Thanks
>>> ___
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>>
>>
>>
>>
>> --
>> Vincenzo Maffione
>>
>
>


-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap with bonded interfaces

[Paras Jha]

It's possible to bypass this by unloading and reloading the patched network
driver

On Thu, Jun 29, 2017 at 12:39 AM, Vincenzo Maffione 
wrote:

> Hi,
>   It is an expected behaviour that you cannot open eth4 and eth5 if they
> are bond, as the device are being used by the lagg pseudo-driver.
> Since this driver does not have netmap support for the moment being, there
> is no way you can get the native mode performance if you use lagg.
>
> If you just need some failover in your application, you could just
> implement a simple failover mechanism in your application (e.g. the
> application opens both netmap:eth4 and netmap:eth5, and decides which one
> to use for transmission depending on which one is up...).
>
> Cheers,
>   Vincenzo
>
> 2017-06-29 4:16 GMT+02:00 Paras Jha :
>
>> Hi all,
>>
>> I have a bonded interface bond0 which enslaves eth4 and eth5. When trying
>> to open the devices eth4 or eth5 via netmap, I get a "device in use"
>> error.
>> Opening the bond0 interface directly in netmap works, however it is in
>> emulated mode (as expected of a pseudointerface)
>>
>> What is the idiomatic way to proceed in such a situation, without
>> compromising on speed?
>>
>> Thanks
>> ___
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>
>
>
>
> --
> Vincenzo Maffione
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap with bonded interfaces

[Vincenzo Maffione]

Hi,
  It is an expected behaviour that you cannot open eth4 and eth5 if they
are bond, as the device are being used by the lagg pseudo-driver.
Since this driver does not have netmap support for the moment being, there
is no way you can get the native mode performance if you use lagg.

If you just need some failover in your application, you could just
implement a simple failover mechanism in your application (e.g. the
application opens both netmap:eth4 and netmap:eth5, and decides which one
to use for transmission depending on which one is up...).

Cheers,
  Vincenzo

2017-06-29 4:16 GMT+02:00 Paras Jha :

> Hi all,
>
> I have a bonded interface bond0 which enslaves eth4 and eth5. When trying
> to open the devices eth4 or eth5 via netmap, I get a "device in use" error.
> Opening the bond0 interface directly in netmap works, however it is in
> emulated mode (as expected of a pseudointerface)
>
> What is the idiomatic way to proceed in such a situation, without
> compromising on speed?
>
> Thanks
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap zero-copy with multiple NICs

[Vincenzo Maffione]

Hi,
  Yes, by default netmap packet buffers for all physical nics on your
machine are allocated from the same memory area. This means that you can do
zcopy with the usual swap of netmap slots.

The bridge application is an example. It does zcopy if possible, otherwise
it falls back to copying.

Cheers,
  Vincenzo


Il 18 apr 2017 1:49 AM, "Paras Jha"  ha scritto:

Is it still possible to share ring buffers across multiple physical network
cards for zero-copy mode, or does the application need to take this into
account and perform a one-copy?
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap pipe zero-copy with NIC buffer?

[Paras Jha]

Thank you for the clarification!

On Wed, Apr 12, 2017 at 3:50 AM, Vincenzo Maffione 
wrote:

> netmap applications always bypass the kernel (network stack). There is
> only a distinction between ports opened in "native" mode and "emulated"
> mode.
> If NIC driver doesn't have native netmap support, the legacy driver is
> internally used by netmap to transmit and receive (and this is called
> "emulated mode").
> In any case (both in native and emulated mode) the netmap application sees
> netmap rings, and can zerocopy between two rings (in the same memory area)
> by simply swapping the ring slots.
>
> Of course, in emulated mode netmap internally transforms each ring slot
> into and mbuf and the other way around, possibly requiring a copy. And the
> whole thing is way slower.
> But this is unrelated to the zerocopy thing: your application isn't aware
> of the emulated or native mode and can zerocopy slots between netmap rings.
>
> Cheers,
>   Vincenzo
>
> 2017-04-12 4:04 GMT+02:00 Paras Jha :
>
>> Apologies, by KB I meant kernel bypass, since it is possible to open a
>> netmap port without bypassing the kernel. I didn't know if this would
>> affect it or not.
>>
>> On Sun, Apr 9, 2017 at 3:20 AM, Vincenzo Maffione 
>> wrote:
>>
>>> Hi,
>>>   Yes, when you nm_open("netmap:em3{2", ...), you're opening a netmap
>>> pipe in the same netmap memory area as the one used by
>>> nm_open("netmap:em3", ...).
>>> As a result, you can zero-copy packets from NIC rings to pipe rings.
>>>
>>> What do you mean by "KB mode"?
>>>
>>> Cheers,
>>>   Vincenzo
>>>
>>> 2017-04-08 19:56 GMT+02:00 Paras Jha :
>>>
 Hi all,

 Is it possible to have a netmap pipe share memory with a netmap port
 opened
 in KB mode for zero-copy purposes?

 All the best
 ___
 freebsd-net@freebsd.org mailing list
 https://lists.freebsd.org/mailman/listinfo/freebsd-net
 To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

>>>
>>>
>>>
>>> --
>>> Vincenzo Maffione
>>>
>>
>>
>
>
> --
> Vincenzo Maffione
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap pipe zero-copy with NIC buffer?

[Vincenzo Maffione]

netmap applications always bypass the kernel (network stack). There is only
a distinction between ports opened in "native" mode and "emulated" mode.
If NIC driver doesn't have native netmap support, the legacy driver is
internally used by netmap to transmit and receive (and this is called
"emulated mode").
In any case (both in native and emulated mode) the netmap application sees
netmap rings, and can zerocopy between two rings (in the same memory area)
by simply swapping the ring slots.

Of course, in emulated mode netmap internally transforms each ring slot
into and mbuf and the other way around, possibly requiring a copy. And the
whole thing is way slower.
But this is unrelated to the zerocopy thing: your application isn't aware
of the emulated or native mode and can zerocopy slots between netmap rings.

Cheers,
  Vincenzo

2017-04-12 4:04 GMT+02:00 Paras Jha :

> Apologies, by KB I meant kernel bypass, since it is possible to open a
> netmap port without bypassing the kernel. I didn't know if this would
> affect it or not.
>
> On Sun, Apr 9, 2017 at 3:20 AM, Vincenzo Maffione 
> wrote:
>
>> Hi,
>>   Yes, when you nm_open("netmap:em3{2", ...), you're opening a netmap
>> pipe in the same netmap memory area as the one used by
>> nm_open("netmap:em3", ...).
>> As a result, you can zero-copy packets from NIC rings to pipe rings.
>>
>> What do you mean by "KB mode"?
>>
>> Cheers,
>>   Vincenzo
>>
>> 2017-04-08 19:56 GMT+02:00 Paras Jha :
>>
>>> Hi all,
>>>
>>> Is it possible to have a netmap pipe share memory with a netmap port
>>> opened
>>> in KB mode for zero-copy purposes?
>>>
>>> All the best
>>> ___
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>>
>>
>>
>>
>> --
>> Vincenzo Maffione
>>
>
>


-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap pipe zero-copy with NIC buffer?

[Paras Jha]

Apologies, by KB I meant kernel bypass, since it is possible to open a
netmap port without bypassing the kernel. I didn't know if this would
affect it or not.

On Sun, Apr 9, 2017 at 3:20 AM, Vincenzo Maffione 
wrote:

> Hi,
>   Yes, when you nm_open("netmap:em3{2", ...), you're opening a netmap pipe
> in the same netmap memory area as the one used by nm_open("netmap:em3",
> ...).
> As a result, you can zero-copy packets from NIC rings to pipe rings.
>
> What do you mean by "KB mode"?
>
> Cheers,
>   Vincenzo
>
> 2017-04-08 19:56 GMT+02:00 Paras Jha :
>
>> Hi all,
>>
>> Is it possible to have a netmap pipe share memory with a netmap port
>> opened
>> in KB mode for zero-copy purposes?
>>
>> All the best
>> ___
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>
>
>
>
> --
> Vincenzo Maffione
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap pipe zero-copy with NIC buffer?

[Vincenzo Maffione]

Hi,
  Yes, when you nm_open("netmap:em3{2", ...), you're opening a netmap pipe
in the same netmap memory area as the one used by nm_open("netmap:em3",
...).
As a result, you can zero-copy packets from NIC rings to pipe rings.

What do you mean by "KB mode"?

Cheers,
  Vincenzo

2017-04-08 19:56 GMT+02:00 Paras Jha :

> Hi all,
>
> Is it possible to have a netmap pipe share memory with a netmap port opened
> in KB mode for zero-copy purposes?
>
> All the best
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap error on -CURRENT with em and igb

[Sean Bruno]

On 02/16/17 02:25, Giuseppe Lettieri wrote:
> Hi all,
> 
> the "Operation not permitted" is coming from iflib_netmap_register:
> 
> ifp->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE);
> ...
> IFDI_INIT(ctx);// for igb it calls em_if_init()
> ...
> return (ifp->if_drv_flags & IFF_DRV_RUNNING ? 0 : 1);
> 
> the last line is meant to check that the initialization was successful,
> but em_if_init() is not setting IFF_DRV_RUNNING.
> 
> Note that the iflib_netmap_register code, and the other netmap register
> functions which it mimicks, have other issues:
> 
> - in case of failure on NIOCREGIF, they leaves the NETMAP_ON flag set
> even if the netmap data structures are deleted;
> - the 1 was intended as a generic indication of failure, but it
> translates to the rather confusing EPERM;
> - they check IFF_DRV_RUNNING flag outside of the lock; I don't know it
> this is safe or not.
> 
> I can prepare a patch fixing these issues, but I don't know wether
> em_if_init() is supposed to set the IFF_DRV_RUNNING flag or not.
> 
> Cheers,
> Giuseppe
> 


With IFLIB, em_if_init() should *not* set IFF_DRV_RUNNING, this should
be done in sys/net/iflib.c

Do you have an idea where you want to put this?

sean



signature.asc
Description: OpenPGP digital signature

Re: netmap error on -CURRENT with em and igb

[Giuseppe Lettieri]

Hi all,

the "Operation not permitted" is coming from iflib_netmap_register:

ifp->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE);
...
IFDI_INIT(ctx); // for igb it calls em_if_init()
...
return (ifp->if_drv_flags & IFF_DRV_RUNNING ? 0 : 1);

the last line is meant to check that the initialization was successful,
but em_if_init() is not setting IFF_DRV_RUNNING.

Note that the iflib_netmap_register code, and the other netmap register 
functions which it mimicks, have other issues:


- in case of failure on NIOCREGIF, they leaves the NETMAP_ON flag set 
even if the netmap data structures are deleted;
- the 1 was intended as a generic indication of failure, but it 
translates to the rather confusing EPERM;
- they check IFF_DRV_RUNNING flag outside of the lock; I don't know it 
this is safe or not.


I can prepare a patch fixing these issues, but I don't know wether 
em_if_init() is supposed to set the IFF_DRV_RUNNING flag or not.


Cheers,
Giuseppe

Il 16/02/2017 09:35, Eggert, Lars ha scritto:

Hi,

I can't put em or igb interfaces into netmap mode on a recent -CURRENT (ix 
interfaces work on the same machines). Here are the pkt-gen and dmesg outputs:

# sudo sysctl dev.netmap.admode=1
# sudo sysctl dev.netmap.verbose=1

# sudo ./pkt-gen -i em1
790.411737 main [2274] interface is em1
790.411753 main [2397] running on 1 cpus (have 20)
790.411823 extract_ip_range [369] range is 10.0.0.1:0 to 10.0.0.1:0
790.411828 extract_ip_range [369] range is 10.1.0.1:0 to 10.1.0.1:0
790.625117 nm_open [898] NIOCREGIF failed: Operation not permitted em1
790.625126 main [2479] Unable to open netmap:em1: Operation not permitted
790.625129 main [2560] aborting

dmesg:
790.411868 [1776] netmap_interp_ringid  em1: tx [0,2) rx [0,2) id 0
790.419431 [ 551] nm_mem_assign_group   iommu_group 0
790.425548 [1148] netmap_config_obj_allocator objsize 1024 clustsize 4096 
objects 4
790.434368 [1148] netmap_config_obj_allocator objsize 36864 clustsize 36864 
objects 1
790.443390 [1148] netmap_config_obj_allocator objsize 2048 clustsize 4096 
objects 2
790.452219 [1270] netmap_finalize_obj_allocator Pre-allocated 25 clusters 
(4/100KB) for 'netmap_if'
790.463605 [1270] netmap_finalize_obj_allocator Pre-allocated 200 clusters 
(36/7200KB) for 'netmap_ring'
790.534919 [1270] netmap_finalize_obj_allocator Pre-allocated 81920 clusters 
(4/327680KB) for 'netmap_buf'
790.546113 [1377] netmap_mem_finalize_all   interfaces 100 KB, rings 7200 KB, 
buffers 320 MB
790.555873 [1380] netmap_mem_finalize_all   Free buffers: 163838
790.605325 [1831] netmap_mem_global_deref   active = 0
790.612967 [ 949] netmap_close  dev 0xf8000a4d7c00 fflag 
0x20003 devtype 8192 td 0xf8000b899000

# sudo ./pkt-gen -i igb1
094.077695 main [2274] interface is igb1
094.077711 main [2397] running on 1 cpus (have 10)
094.077822 extract_ip_range [369] range is 10.0.0.1:0 to 10.0.0.1:0
094.077827 extract_ip_range [369] range is 10.1.0.1:0 to 10.1.0.1:0
094.280311 nm_open [898] NIOCREGIF failed: Operation not permitted igb1
094.280319 main [2479] Unable to open netmap:igb1: Operation not permitted
094.280323 main [2560] aborting

dmesg:
094.077866 [1776] netmap_interp_ringid  igb1: tx [0,8) rx [0,8) id 0
094.085425 [ 551] nm_mem_assign_group   iommu_group 0
094.091449 [1148] netmap_config_obj_allocator objsize 1024 clustsize 4096 
objects 4
094.100125 [1148] netmap_config_obj_allocator objsize 36864 clustsize 36864 
objects 1
094.109004 [1148] netmap_config_obj_allocator objsize 2048 clustsize 4096 
objects 2
094.117705 [1270] netmap_finalize_obj_allocator Pre-allocated 25 clusters 
(4/100KB) for 'netmap_if'
094.128918 [1270] netmap_finalize_obj_allocator Pre-allocated 200 clusters 
(36/7200KB) for 'netmap_ring'
094.199456 [1270] netmap_finalize_obj_allocator Pre-allocated 81920 clusters 
(4/327680KB) for 'netmap_buf'
094.210482 [1377] netmap_mem_finalize_all   interfaces 100 KB, rings 7200 KB, 
buffers 320 MB
094.220078 [1380] netmap_mem_finalize_all   Free buffers: 163838
094.260201 [1831] netmap_mem_global_deref   active = 0
094.268364 [ 949] netmap_close  dev 0xf8000a453000 fflag 
0x20003 devtype 8192 td 0xf80014d45000

Lars




--
Dr. Ing. Giuseppe Lettieri
Dipartimento di Ingegneria della Informazione
Universita' di Pisa
Largo Lucio Lazzarino 1, 56122 Pisa - Italy
Ph. : (+39) 050-2217.649 (direct) .599 (switch)
Fax : (+39) 050-2217.600
e-mail: g.letti...@iet.unipi.it
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap TX with no impact to host

[Vincenzo Maffione]

You're welcome! I know it's a bit difficult to notice, but this is written
in the very last sentence of the netmap(4) man page.

Cheers,
  Vincenzo

2017-01-19 13:50 GMT+01:00 David Belle-Isle :

> Oh! Thank you s much! I was missing the last part about the checksum
> offloadings and TSO. I disabled them and everything is working perfectly
> now! Thank you so much!!
>
> On Thu, Jan 19, 2017 at 6:00 AM, Vincenzo Maffione 
> wrote:
>
>> Hi,
>>   Before answering to the question, some important disclaimers.
>>
>> If you are using FreeBSD-12-current, please be aware that there are IFLIB
>> developments in progress that affect em devices. Unfortunately, at the
>> moment netmap does not work on em/lem/igb devices because of the switch to
>> IFLIB at this commit https://github.com/freebsd/fre
>> ebsd/commit/4bf01b136dfa554a57f25559e7a848bf32206c66. Netmap on em works
>> correctly right before this commit.
>>
>> If you are using FreeBSD-11 versions, be aware that netmap in emulated
>> mode is broken in those versions. I have the fix for that, but it is not in
>> the released code. If you need it, I can give you a patch. Btw, can anyone
>> point me at the procedure to push this fix to the relevant release/stable
>> branches?
>> If you use em devices with DEV_NETMAP enabled, emulated mode is not used,
>> so it will work. If you use interfaces without DEV_NETMAP support, emulated
>> netmap is used and you are affected by the broken code.
>>
>> Regarding your question:
>> You can build a netmap program that does what you want:
>> (1) transmit packets on em0 at high rates using netmap
>> (2) let the host stack transmit packets on em0
>> (3) let packets received on em0 to find their way to the host stack.
>>
>> In netmap it is possible to open just a subset of the TX rings and/or a
>> subset of the RX rings.
>> In your case you need to open all the TX hw rings (to acheve (1)) and the
>> RX host ring (a.k.a. RX sw ring) to achieve (2).
>> You achieve (3) by simply not opening the RX hw rings.
>> To implement this you need two calls to nm_open() (similarly to what
>> bridge does):
>>
>> pa = nm_open("netmap:em0/T, NULL, 0, NULL); /* open all TX hw rings */
>> pb = nm_open("netmap:em0^/R, NULL, NM_OPEN_NO_MMAP, pa); /* open only RX
>> host ring */
>>
>> Then you can use "pa" to transmit packets on the hw TX rings (1).
>> To implement (2), you have two options:
>>   (A) your application moves packets from the host RX ring (pb) to the hw
>> TX rings (pa). In the end this is the logic of the "bridge" program does,
>> so you could start from there.
>>   (B) you use "transparent mode" in the host->nic direction: applications
>> marks with the NS_FORWARD flag the host RX ring slots right before they
>> released to the kernel, and the kernel with take care of forward them to
>> the TX ring for you. However, be aware this mode is still experimental, and
>> not well tested yet.
>>
>> Regarding the use of bridge, the correct command line for what you want
>> to do (forward all packets between host stack and NIC, in both directions)
>> is:
>>
>> # bridge netmap:em0
>>
>> However, to let TCP/UDP traffic go correctly, you need to disable tso and
>> checksum offloadings on em0, since netmap will not program the NIC to
>> perform those offloadings. This is also valid for your custom program above.
>>
>> Cheers,
>>   Vincenzo
>>
>> 2017-01-19 2:51 GMT+01:00 David Belle-Isle :
>>
>>> Hi,
>>>
>>> I'm trying to open a netmap descriptor to an interface to send packets.
>>> However, I don't want to prevent the host to send and receive data
>>> (transparent). I don't think this should be hard but I can't figure out
>>> how
>>> to do this.
>>>
>>> I tried to run the bridge example in the FreeBSD distribution but even
>>> that
>>> I can't get to run without stopping the host's networking completely. I
>>> tried running ./bridge em0 em0 which, if I understand correctly should
>>> open
>>> the NIC and host rings and pass the traffic through. As soon as I start
>>> it
>>> all the networking stops.
>>>
>>> I tried testing in a VMware VM and on bare-metal with em cards and got
>>> the
>>> same results with both.
>>>
>>> Can someone help me?
>>>
>>> Thanks,
>>>
>>> David
>>> ___
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>>
>>
>>
>>
>> --
>> Vincenzo Maffione
>>
>
>


-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap TX with no impact to host

[David Belle-Isle]

Oh! Thank you s much! I was missing the last part about the checksum
offloadings and TSO. I disabled them and everything is working perfectly
now! Thank you so much!!

On Thu, Jan 19, 2017 at 6:00 AM, Vincenzo Maffione 
wrote:

> Hi,
>   Before answering to the question, some important disclaimers.
>
> If you are using FreeBSD-12-current, please be aware that there are IFLIB
> developments in progress that affect em devices. Unfortunately, at the
> moment netmap does not work on em/lem/igb devices because of the switch to
> IFLIB at this commit https://github.com/freebsd/freebsd/commit/
> 4bf01b136dfa554a57f25559e7a848bf32206c66. Netmap on em works correctly
> right before this commit.
>
> If you are using FreeBSD-11 versions, be aware that netmap in emulated
> mode is broken in those versions. I have the fix for that, but it is not in
> the released code. If you need it, I can give you a patch. Btw, can anyone
> point me at the procedure to push this fix to the relevant release/stable
> branches?
> If you use em devices with DEV_NETMAP enabled, emulated mode is not used,
> so it will work. If you use interfaces without DEV_NETMAP support, emulated
> netmap is used and you are affected by the broken code.
>
> Regarding your question:
> You can build a netmap program that does what you want:
> (1) transmit packets on em0 at high rates using netmap
> (2) let the host stack transmit packets on em0
> (3) let packets received on em0 to find their way to the host stack.
>
> In netmap it is possible to open just a subset of the TX rings and/or a
> subset of the RX rings.
> In your case you need to open all the TX hw rings (to acheve (1)) and the
> RX host ring (a.k.a. RX sw ring) to achieve (2).
> You achieve (3) by simply not opening the RX hw rings.
> To implement this you need two calls to nm_open() (similarly to what
> bridge does):
>
> pa = nm_open("netmap:em0/T, NULL, 0, NULL); /* open all TX hw rings */
> pb = nm_open("netmap:em0^/R, NULL, NM_OPEN_NO_MMAP, pa); /* open only RX
> host ring */
>
> Then you can use "pa" to transmit packets on the hw TX rings (1).
> To implement (2), you have two options:
>   (A) your application moves packets from the host RX ring (pb) to the hw
> TX rings (pa). In the end this is the logic of the "bridge" program does,
> so you could start from there.
>   (B) you use "transparent mode" in the host->nic direction: applications
> marks with the NS_FORWARD flag the host RX ring slots right before they
> released to the kernel, and the kernel with take care of forward them to
> the TX ring for you. However, be aware this mode is still experimental, and
> not well tested yet.
>
> Regarding the use of bridge, the correct command line for what you want to
> do (forward all packets between host stack and NIC, in both directions) is:
>
> # bridge netmap:em0
>
> However, to let TCP/UDP traffic go correctly, you need to disable tso and
> checksum offloadings on em0, since netmap will not program the NIC to
> perform those offloadings. This is also valid for your custom program above.
>
> Cheers,
>   Vincenzo
>
> 2017-01-19 2:51 GMT+01:00 David Belle-Isle :
>
>> Hi,
>>
>> I'm trying to open a netmap descriptor to an interface to send packets.
>> However, I don't want to prevent the host to send and receive data
>> (transparent). I don't think this should be hard but I can't figure out
>> how
>> to do this.
>>
>> I tried to run the bridge example in the FreeBSD distribution but even
>> that
>> I can't get to run without stopping the host's networking completely. I
>> tried running ./bridge em0 em0 which, if I understand correctly should
>> open
>> the NIC and host rings and pass the traffic through. As soon as I start it
>> all the networking stops.
>>
>> I tried testing in a VMware VM and on bare-metal with em cards and got the
>> same results with both.
>>
>> Can someone help me?
>>
>> Thanks,
>>
>> David
>> ___
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>
>
>
>
> --
> Vincenzo Maffione
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap TX with no impact to host

[Vincenzo Maffione]

Hi,
  Before answering to the question, some important disclaimers.

If you are using FreeBSD-12-current, please be aware that there are IFLIB
developments in progress that affect em devices. Unfortunately, at the
moment netmap does not work on em/lem/igb devices because of the switch to
IFLIB at this commit
https://github.com/freebsd/freebsd/commit/4bf01b136dfa554a57f25559e7a848bf32206c66.
Netmap on em works correctly right before this commit.

If you are using FreeBSD-11 versions, be aware that netmap in emulated mode
is broken in those versions. I have the fix for that, but it is not in the
released code. If you need it, I can give you a patch. Btw, can anyone
point me at the procedure to push this fix to the relevant release/stable
branches?
If you use em devices with DEV_NETMAP enabled, emulated mode is not used,
so it will work. If you use interfaces without DEV_NETMAP support, emulated
netmap is used and you are affected by the broken code.

Regarding your question:
You can build a netmap program that does what you want:
(1) transmit packets on em0 at high rates using netmap
(2) let the host stack transmit packets on em0
(3) let packets received on em0 to find their way to the host stack.

In netmap it is possible to open just a subset of the TX rings and/or a
subset of the RX rings.
In your case you need to open all the TX hw rings (to acheve (1)) and the
RX host ring (a.k.a. RX sw ring) to achieve (2).
You achieve (3) by simply not opening the RX hw rings.
To implement this you need two calls to nm_open() (similarly to what bridge
does):

pa = nm_open("netmap:em0/T, NULL, 0, NULL); /* open all TX hw rings */
pb = nm_open("netmap:em0^/R, NULL, NM_OPEN_NO_MMAP, pa); /* open only RX
host ring */

Then you can use "pa" to transmit packets on the hw TX rings (1).
To implement (2), you have two options:
  (A) your application moves packets from the host RX ring (pb) to the hw
TX rings (pa). In the end this is the logic of the "bridge" program does,
so you could start from there.
  (B) you use "transparent mode" in the host->nic direction: applications
marks with the NS_FORWARD flag the host RX ring slots right before they
released to the kernel, and the kernel with take care of forward them to
the TX ring for you. However, be aware this mode is still experimental, and
not well tested yet.

Regarding the use of bridge, the correct command line for what you want to
do (forward all packets between host stack and NIC, in both directions) is:

# bridge netmap:em0

However, to let TCP/UDP traffic go correctly, you need to disable tso and
checksum offloadings on em0, since netmap will not program the NIC to
perform those offloadings. This is also valid for your custom program above.

Cheers,
  Vincenzo

2017-01-19 2:51 GMT+01:00 David Belle-Isle :

> Hi,
>
> I'm trying to open a netmap descriptor to an interface to send packets.
> However, I don't want to prevent the host to send and receive data
> (transparent). I don't think this should be hard but I can't figure out how
> to do this.
>
> I tried to run the bridge example in the FreeBSD distribution but even that
> I can't get to run without stopping the host's networking completely. I
> tried running ./bridge em0 em0 which, if I understand correctly should open
> the NIC and host rings and pass the traffic through. As soon as I start it
> all the networking stops.
>
> I tried testing in a VMware VM and on bare-metal with em cards and got the
> same results with both.
>
> Can someone help me?
>
> Thanks,
>
> David
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>



-- 
Vincenzo Maffione
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: netmap, netmap-fwd, and how many M packets-per-second?

[Vincenzo Maffione]

Hi,
  If I'm getting it right, you are doing a test with 10 machines directly
connected on the same LAN, using iperf to test the performance of the
TCP/IP stack with your 10Gbit cards.
But as far as I can see, netmap is not involved in your setup, you are not
using it, aren't you?
IOW , I don't understand what does that "+ Netmap" mean you our test
description banner.

Cheers,
  Vincenzo

2016-12-02 0:55 GMT+01:00 Jordan Caraballo :

> Feedback and/or tips and tricks more than welcome.
>
> We are trying to process huge amounts of small (64 bytes) pps through a
> router. So far results have not been as we expected. We have tested
> FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without
> netmap. Based on netmap documentation we were expecting about 5.0M pps;
> alongside with the routing improvements from the freebsd routing
> proposal a total of 12.0M.
>
> Server Description:
>
> Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5­2695 CPU's, 18 cores per
> cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot.
>
> BIOS tweaks:
>
> Hyperthreading (or Logical Processors) is turned off.
>
> Current results are shown below. Additional configurations can be given
> upon request.
>
> Test Environment:
> 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one
> equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2).
>
> Script that execute the following on each host.
>
> #!/usr/local/bin/bash
> # Iterate through ports and start tests
> for ((i=1;i<=PORTS;i++)); do
> PORT=$(($PORT+1))
> iperf3 -c 172.16.2.$IP -u -b1m -i 0 -N -l$PKT -t$TIME -P$STREAMS
> -p$PORT &
> #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT &
> done
>
> # FreeBSD 10.3 - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.9M 0  1.3M   194M   540k 057M 0 0
>   2.1M 0  1.5M   216M   556k 058M 0 0
>   1.8M 0  1.3M   192M   553k 058M 0 0
>   1.7M 0  1.1M   174M   542k 057M 0 0
>   1.9M 0  1.4M   204M   537k 056M 0 0
>   1.6M 0  1.1M   171M   550k 058M 0 0
>   1.6M 0  1.1M   173M   546k 057M 0 0
>   1.7M 0  1.1M   176M   564k 059M 0 0
>   2.0M 0  1.5M   212M   543k 057M 0 0
>   2.1M 0  1.5M   219M   557k 058M 0 0
>   1.9M 0  1.4M   205M   547k 057M 0 0
>   1.7M 0  1.2M   179M   553k 058M 0 0
>
> # FreeBSD 11.0 - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   3.1M 0  1.8M   326M   1.3M 0   134M 0 0
>   2.6M 0  1.5M   269M   1.1M 0   116M 0 0
>   2.7M 0  1.5M   285M   1.2M 0   127M 0 0
>   2.4M 0  1.3M   257M   1.1M 0   119M 0 0
>   2.7M 0  1.5M   287M   1.3M 0   134M 0 0
>   2.5M 0  1.3M   262M   1.2M 0   127M 0 0
>   2.1M 0  1.1M   224M   1.0M 0   108M 0 0
>   2.7M 0  1.4M   285M   1.4M 0   143M 0 0
>   2.6M 0  1.3M   272M   1.3M 0   136M 0 0
>   2.5M 0  1.4M   265M   1.1M 0   120M 0 0
> # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.9M 0  849k   195M   1.0M 0   107M 0 0
>   1.9M 0  854k   196M   1.0M 0   106M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 0 0
>   1.9M 0  852k   196M   1.0M 0   107M 0 0
>   1.9M 0  847k   195M   1.0M 0   107M 0 0
>   1.9M 0  836k   195M   1.0M 0   107M 0 0
>   1.9M 0  843k   195M   1.0M 0   107M 0 0
> # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5)
>
>input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.1M   259 0   115M   1.1M 0   115M 0 0
>   1.2M   273 0   124M   1.2M 0   124M 0 0
>   1.1M   200 0   112M   1.1M 0   112M 0 0
>   1.2M   290   

Re: netmap, netmap-fwd, and how many M packets-per-second?

[Jakub Palider]

Hi,

I have run some performance tests but not as router, which is your case, so
some information might not be relevant:
0. validate your hw stats with what you get from the output
1. Try using FLOWTABLE option in your kernel config - this may kill your
machine in -CURRENT, but it's worth checking
2. MTU set to some reasonably small value (~120) shall prevent from
consolidating small packets into big frame
3. disable all kernel tracing (INVARIANTS, WITNESS, etc)
4. I read you tuned the system, but make sure there are enough resources
5. Check that context switches are not killing it

Regardless on the above you might want profile the system with dtrace and
export results to flamegraph, this will give you an idea what going on in
the system and possibly locate the cause of the problem. It's definitely
worth the effort.

Jakub

On Fri, Dec 2, 2016 at 12:55 AM, Jordan Caraballo <
jordancaraball...@gmail.com> wrote:

> Feedback and/or tips and tricks more than welcome.
>
> We are trying to process huge amounts of small (64 bytes) pps through a
> router. So far results have not been as we expected. We have tested
> FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without
> netmap. Based on netmap documentation we were expecting about 5.0M pps;
> alongside with the routing improvements from the freebsd routing
> proposal a total of 12.0M.
>
> Server Description:
>
> Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5­2695 CPU's, 18 cores per
> cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot.
>
> BIOS tweaks:
>
> Hyperthreading (or Logical Processors) is turned off.
>
> Current results are shown below. Additional configurations can be given
> upon request.
>
> Test Environment:
> 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one
> equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2).
>
> Script that execute the following on each host.
>
> #!/usr/local/bin/bash
> # Iterate through ports and start tests
> for ((i=1;i<=PORTS;i++)); do
> PORT=$(($PORT+1))
> iperf3 -c 172.16.2.$IP -u -b1m -i 0 -N -l$PKT -t$TIME -P$STREAMS
> -p$PORT &
> #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT &
> done
>
> # FreeBSD 10.3 - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.9M 0  1.3M   194M   540k 057M 0 0
>   2.1M 0  1.5M   216M   556k 058M 0 0
>   1.8M 0  1.3M   192M   553k 058M 0 0
>   1.7M 0  1.1M   174M   542k 057M 0 0
>   1.9M 0  1.4M   204M   537k 056M 0 0
>   1.6M 0  1.1M   171M   550k 058M 0 0
>   1.6M 0  1.1M   173M   546k 057M 0 0
>   1.7M 0  1.1M   176M   564k 059M 0 0
>   2.0M 0  1.5M   212M   543k 057M 0 0
>   2.1M 0  1.5M   219M   557k 058M 0 0
>   1.9M 0  1.4M   205M   547k 057M 0 0
>   1.7M 0  1.2M   179M   553k 058M 0 0
>
> # FreeBSD 11.0 - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   3.1M 0  1.8M   326M   1.3M 0   134M 0 0
>   2.6M 0  1.5M   269M   1.1M 0   116M 0 0
>   2.7M 0  1.5M   285M   1.2M 0   127M 0 0
>   2.4M 0  1.3M   257M   1.1M 0   119M 0 0
>   2.7M 0  1.5M   287M   1.3M 0   134M 0 0
>   2.5M 0  1.3M   262M   1.2M 0   127M 0 0
>   2.1M 0  1.1M   224M   1.0M 0   108M 0 0
>   2.7M 0  1.4M   285M   1.4M 0   143M 0 0
>   2.6M 0  1.3M   272M   1.3M 0   136M 0 0
>   2.5M 0  1.4M   265M   1.1M 0   120M 0 0
> # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.9M 0  849k   195M   1.0M 0   107M 0 0
>   1.9M 0  854k   196M   1.0M 0   106M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 0 0
>   1.9M 0  852k   196M   1.0M 0   107M 0 0
>   1.9M 0  847k   195M   1.0M 0   107M 0 0
>   1.9M 0  836k   195M   1.0M 0   107M 0 0
>   1.9M 0  843k   195M   1.0M   

Re: netmap, netmap-fwd, and how many M packets-per-second?

[Navdeep Parhar]

How have you configured netmap-fwd?  If you provide details on how the
router or firewall is setup I can try similar experiments here.

On Thu, Dec 1, 2016 at 3:55 PM, Jordan Caraballo
 wrote:
> Feedback and/or tips and tricks more than welcome.
>
> We are trying to process huge amounts of small (64 bytes) pps through a
> router. So far results have not been as we expected. We have tested
> FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without
> netmap. Based on netmap documentation we were expecting about 5.0M pps;
> alongside with the routing improvements from the freebsd routing
> proposal a total of 12.0M.
>
> Server Description:
>
> Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5­2695 CPU's, 18 cores per
> cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot.
>
> BIOS tweaks:
>
> Hyperthreading (or Logical Processors) is turned off.
>
> Current results are shown below. Additional configurations can be given
> upon request.
>
> Test Environment:
> 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one
> equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2).
>
> Script that execute the following on each host.
>
> #!/usr/local/bin/bash
> # Iterate through ports and start tests
> for ((i=1;i<=PORTS;i++)); do
> PORT=$(($PORT+1))
> iperf3 -c 172.16.2.$IP -u -b1m -i 0 -N -l$PKT -t$TIME -P$STREAMS
> -p$PORT &
> #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT &
> done
>
> # FreeBSD 10.3 - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.9M 0  1.3M   194M   540k 057M 0 0
>   2.1M 0  1.5M   216M   556k 058M 0 0
>   1.8M 0  1.3M   192M   553k 058M 0 0
>   1.7M 0  1.1M   174M   542k 057M 0 0
>   1.9M 0  1.4M   204M   537k 056M 0 0
>   1.6M 0  1.1M   171M   550k 058M 0 0
>   1.6M 0  1.1M   173M   546k 057M 0 0
>   1.7M 0  1.1M   176M   564k 059M 0 0
>   2.0M 0  1.5M   212M   543k 057M 0 0
>   2.1M 0  1.5M   219M   557k 058M 0 0
>   1.9M 0  1.4M   205M   547k 057M 0 0
>   1.7M 0  1.2M   179M   553k 058M 0 0
>
> # FreeBSD 11.0 - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   3.1M 0  1.8M   326M   1.3M 0   134M 0 0
>   2.6M 0  1.5M   269M   1.1M 0   116M 0 0
>   2.7M 0  1.5M   285M   1.2M 0   127M 0 0
>   2.4M 0  1.3M   257M   1.1M 0   119M 0 0
>   2.7M 0  1.5M   287M   1.3M 0   134M 0 0
>   2.5M 0  1.3M   262M   1.2M 0   127M 0 0
>   2.1M 0  1.1M   224M   1.0M 0   108M 0 0
>   2.7M 0  1.4M   285M   1.4M 0   143M 0 0
>   2.6M 0  1.3M   272M   1.3M 0   136M 0 0
>   2.5M 0  1.4M   265M   1.1M 0   120M 0 0
> # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.9M 0  849k   195M   1.0M 0   107M 0 0
>   1.9M 0  854k   196M   1.0M 0   106M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 0 0
>   1.9M 0  852k   196M   1.0M 0   107M 0 0
>   1.9M 0  847k   195M   1.0M 0   107M 0 0
>   1.9M 0  836k   195M   1.0M 0   107M 0 0
>   1.9M 0  843k   195M   1.0M 0   107M 0 0
> # FreeBSD 12.0-CURRENT - 4 streams to 80 ports from each client (5)
>
>input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.1M   259 0   115M   1.1M 0   115M 0 0
>   1.2M   273 0   124M   1.2M 0   124M 0 0
>   1.1M   200 0   112M   1.1M 0   112M 0 0
>   1.2M   290 0   122M   1.2M 0   122M 0 0
>   1.0M   132 0   107M   1.0M 0   107M 0 0
>   1.1M   303 0   118M   1.1M 0   118M 0 0
>   1.1M   278 0   112M   1.1M

Re: netmap, netmap-fwd, and how many M packets-per-second?

[Jim Thompson]

(I'm not subscribed to -hpc or -performance, so I've trimmed the
recipients.)

You're running iperf3 on an Ivy Bridge Xeon at 2.4GHz.

-N (--no-delay) only applies to TCP, it disables Nagle's algorithm, so it
doesn't apply for "-u" (--udp).

In any case, iperf3 still attempts to use large enough frames to be able to
fill the bandwidth limit (you've requested 'b1m' = 10Gbps) with a
minimum number of packets.  It does this by writing as much data as will
fit in the sockbuf (if it gets back EWOULDBLOCK) in a loop.

https://github.com/esnet/iperf/blob/099244ec686b620393e9845478a554b1c7ca5c8b/src/net.c#L251

A full sized ethernet frame is 1538 bytes on the wire (without any 802.1q
tags).
1,000,000,000 b/s / (1,538 B * 8 b/B) == 81,274 pps
10G is 812,743pps
40G is 3,250,975pps

and you're quite near this (3.2Mpps) with the  FreeBSD 11.0 setup.  Are you
sure you're not running a WITNESS/DEBUG kernel on 12-CURRENT?

The (version of) netmap-fwd you have is (still) single-threaded, so you get
one core.
You probably also don't have the hw cksum offload parts for netmap that we
did.

You're seeing about 1.5Mpps and at 1538 byte frames (on the wire) that's
around 18.5Gbps.

try pkt_gen, it will generate the small frames you seek.

Cheers,
Jim


On Thu, Dec 1, 2016 at 5:55 PM, Jordan Caraballo <
jordancaraball...@gmail.com> wrote:

> Feedback and/or tips and tricks more than welcome.
>
> We are trying to process huge amounts of small (64 bytes) pps through a
> router. So far results have not been as we expected. We have tested
> FreeBSD 10.3, 11.0, 11.0-STABLE, and 12.0-CURRENT with and without
> netmap. Based on netmap documentation we were expecting about 5.0M pps;
> alongside with the routing improvements from the freebsd routing
> proposal a total of 12.0M.
>
> Server Description:
>
> Dell PowerEdge R530 with 2 Intel(R) Xeon(R) E5­2695 CPU's, 18 cores per
> cpu. Equipped with a Chelsio T-580-CR dual port in an 8x slot.
>
> BIOS tweaks:
>
> Hyperthreading (or Logical Processors) is turned off.
>
> Current results are shown below. Additional configurations can be given
> upon request.
>
> Test Environment:
> 5 clients and 5 servers - 4 Dell C6100 and 2 Dell R420; each one
> equipped with 10G NICS (4 intel 8259X and 6 with mellanox connectx2).
>
> Script that execute the following on each host.
>
> #!/usr/local/bin/bash
> # Iterate through ports and start tests
> for ((i=1;i<=PORTS;i++)); do
> PORT=$(($PORT+1))
> iperf3 -c 172.16.2.$IP -u -b1m -i 0 -N -l$PKT -t$TIME -P$STREAMS
> -p$PORT &
> #iperf3 -c 172.16.2.$IP -i0 -N -l$PKT -t$TIME -P$STREAMS -p$PORT &
> done
>
> # FreeBSD 10.3 - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.9M 0  1.3M   194M   540k 057M 0 0
>   2.1M 0  1.5M   216M   556k 058M 0 0
>   1.8M 0  1.3M   192M   553k 058M 0 0
>   1.7M 0  1.1M   174M   542k 057M 0 0
>   1.9M 0  1.4M   204M   537k 056M 0 0
>   1.6M 0  1.1M   171M   550k 058M 0 0
>   1.6M 0  1.1M   173M   546k 057M 0 0
>   1.7M 0  1.1M   176M   564k 059M 0 0
>   2.0M 0  1.5M   212M   543k 057M 0 0
>   2.1M 0  1.5M   219M   557k 058M 0 0
>   1.9M 0  1.4M   205M   547k 057M 0 0
>   1.7M 0  1.2M   179M   553k 058M 0 0
>
> # FreeBSD 11.0 - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   3.1M 0  1.8M   326M   1.3M 0   134M 0 0
>   2.6M 0  1.5M   269M   1.1M 0   116M 0 0
>   2.7M 0  1.5M   285M   1.2M 0   127M 0 0
>   2.4M 0  1.3M   257M   1.1M 0   119M 0 0
>   2.7M 0  1.5M   287M   1.3M 0   134M 0 0
>   2.5M 0  1.3M   262M   1.2M 0   127M 0 0
>   2.1M 0  1.1M   224M   1.0M 0   108M 0 0
>   2.7M 0  1.4M   285M   1.4M 0   143M 0 0
>   2.6M 0  1.3M   272M   1.3M 0   136M 0 0
>   2.5M 0  1.4M   265M   1.1M 0   120M 0 0
> # FreeBSD 11.0-STABLE - 4 streams to 80 ports from each client (5)
>
> input(Total)   output
>packets  errs idrops  bytespackets  errs  bytes colls drops
>   1.9M 0  849k   195M   1.0M 0   107M 0 0
>   1.9M 0  854k   196M   1.0M 0   106M 0 0
>   1.9M 0  851k   196M   1.0M 0   107M 

Re: NETMAP application architecture question

[Slawa Olhovchenkov]

On Fri, Sep 16, 2016 at 05:05:33PM +0200, Luigi Rizzo wrote:

> Hi,
> I don't think there is any hard limitation on the total number
> of pipes, very likely the number (64) is just some arbitrary
> limit that we set, and can be increased through a sysctl or
> worst case a compile time constant.
> 
> Please have a look at the source code if you can find where
> you are hitting the limit, and which parameters are involved.
> In case you don't find it, get back to us in a week (we are
> busy on a deadline at the moment).
> 
> Note that you should look at the netmap code on github, the one in
> the freebsd tree is not up to date.

Next about tilera-like.
Can I allocate some buffers for custom-made pools?
I.e. balancer thread do some like:

src = _ring->slot[rxr->cur];
free_buf_idx = next_free(private_pool);
to_worker.idx = src->buf_idx;
to_worker.len = src->len;
src->buf_idx = tmp;
src->flags = NS_BUF_CHANGED;
rxr->head = rxr->cur = nm_ring_next(rxr, rxr->cur);

to_worker pass to worker w/o netmap.
balancer put to_worker in to private_pool after worker done.

I am need reserved about x64 of one ring size per balancer.
i.e. about 131072 buffers per balancer and about 512K total buffers.
How I can do this?

> On Fri, Sep 16, 2016 at 4:57 PM, Slawa Olhovchenkov  wrote:
> > I am wrote NETMAP application and meet some limitation.
> > I am use server 16-core server.
> > Workers thread run mostly independend, with minimal interworking.
> > I am use balancer threads for distributing packet flow between worker
> > threads.
> > I am using two NETMAP pipes between balancer and workers.
> > As result, for case of 4 balancer and 12 workers I am need 4x12x2 = 96
> > pipes. NETMAP have limitation of 64 pipes.
> >
> > May be exist more optimal way?
> >
> > May be I can emulate tilera way, with bufers credits?
> > (In this case balancer give RX buffers, confirm to kernel and pass it
> > to worker. kernel reduce credit, used for reciving other buffer from
> > pool. After complete worker pass buffer to balancer and balancer
> > return buffer to kernel, kernel rise credits and put free buffer to
> > pool).
> > In this case I am not need to many pipes and not limited performance
> > by too long processing of one packet.
> > ___
> > freebsd-net@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
> 
> 
> 
> -- 
> -+---
>  Prof. Luigi RIZZO, ri...@iet.unipi.it  . Dip. di Ing. dell'Informazione
>  http://www.iet.unipi.it/~luigi/. Universita` di Pisa
>  TEL  +39-050-2217533   . via Diotisalvi 2
>  Mobile   +39-338-6809875   . 56122 PISA (Italy)
> -+---
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: NETMAP application architecture question

[Michael Shirk]

I am "VERY WILLING" to test out a specific branch of FreeBSD current with
this code merged. There are wonderful features such as the new LB app in
the new code that I would greatly desire to be included in FreeBSD.

I tried to do a simple merge and it was more involved. I was hoping to have
some time to look at the code to try to figure it out.

--
Michael Shirk
Daemon Security, Inc.
http://www.daemon-security.com

On Sep 16, 2016 11:20 AM, "Luigi Rizzo"  wrote:

> On Fri, Sep 16, 2016 at 5:14 PM, Slawa Olhovchenkov 
> wrote:
> > On Fri, Sep 16, 2016 at 05:05:33PM +0200, Luigi Rizzo wrote:
> >
>
> >
> >> Please have a look at the source code if you can find where
> >> you are hitting the limit, and which parameters are involved.
> >> In case you don't find it, get back to us in a week (we are
> >> busy on a deadline at the moment).
> >
> > OK
> >
> >> Note that you should look at the netmap code on github, the one in
> >> the freebsd tree is not up to date.
> >
> > Do you plan to update code in freebsd tree?
>
> yes but don't hold your breath for that, since I haven't found
> time for a while due to other committments.
>
> As a matter of fact if some committer has time and interest
> to test the new code and push it into the tree I am happy to help.
>
> cheers
> luigi
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: NETMAP application architecture question

[Luigi Rizzo]

On Fri, Sep 16, 2016 at 5:14 PM, Slawa Olhovchenkov  wrote:
> On Fri, Sep 16, 2016 at 05:05:33PM +0200, Luigi Rizzo wrote:
>

>
>> Please have a look at the source code if you can find where
>> you are hitting the limit, and which parameters are involved.
>> In case you don't find it, get back to us in a week (we are
>> busy on a deadline at the moment).
>
> OK
>
>> Note that you should look at the netmap code on github, the one in
>> the freebsd tree is not up to date.
>
> Do you plan to update code in freebsd tree?

yes but don't hold your breath for that, since I haven't found
time for a while due to other committments.

As a matter of fact if some committer has time and interest
to test the new code and push it into the tree I am happy to help.

cheers
luigi
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: NETMAP application architecture question

[Slawa Olhovchenkov]

On Fri, Sep 16, 2016 at 05:05:33PM +0200, Luigi Rizzo wrote:

> Hi,
> I don't think there is any hard limitation on the total number
> of pipes, very likely the number (64) is just some arbitrary
> limit that we set, and can be increased through a sysctl or
> worst case a compile time constant.

I am don't see sysctl for this.
Re-compile is not comfortable.

In any way, for this setup I am need to many resources, as I am calc:
buf_num=1114112
ring_num=964
if_num=248

This can introduce to high overhead.

> Please have a look at the source code if you can find where
> you are hitting the limit, and which parameters are involved.
> In case you don't find it, get back to us in a week (we are
> busy on a deadline at the moment).

OK

> Note that you should look at the netmap code on github, the one in
> the freebsd tree is not up to date.

Do you plan to update code in freebsd tree?

> Cheers
> Luigi
> 
> On Fri, Sep 16, 2016 at 4:57 PM, Slawa Olhovchenkov  wrote:
> > I am wrote NETMAP application and meet some limitation.
> > I am use server 16-core server.
> > Workers thread run mostly independend, with minimal interworking.
> > I am use balancer threads for distributing packet flow between worker
> > threads.
> > I am using two NETMAP pipes between balancer and workers.
> > As result, for case of 4 balancer and 12 workers I am need 4x12x2 = 96
> > pipes. NETMAP have limitation of 64 pipes.
> >
> > May be exist more optimal way?
> >
> > May be I can emulate tilera way, with bufers credits?
> > (In this case balancer give RX buffers, confirm to kernel and pass it
> > to worker. kernel reduce credit, used for reciving other buffer from
> > pool. After complete worker pass buffer to balancer and balancer
> > return buffer to kernel, kernel rise credits and put free buffer to
> > pool).
> > In this case I am not need to many pipes and not limited performance
> > by too long processing of one packet.
> > ___
> > freebsd-net@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
> 
> 
> 
> -- 
> -+---
>  Prof. Luigi RIZZO, ri...@iet.unipi.it  . Dip. di Ing. dell'Informazione
>  http://www.iet.unipi.it/~luigi/. Universita` di Pisa
>  TEL  +39-050-2217533   . via Diotisalvi 2
>  Mobile   +39-338-6809875   . 56122 PISA (Italy)
> -+---
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: NETMAP application architecture question

[Luigi Rizzo]

Hi,
I don't think there is any hard limitation on the total number
of pipes, very likely the number (64) is just some arbitrary
limit that we set, and can be increased through a sysctl or
worst case a compile time constant.

Please have a look at the source code if you can find where
you are hitting the limit, and which parameters are involved.
In case you don't find it, get back to us in a week (we are
busy on a deadline at the moment).

Note that you should look at the netmap code on github, the one in
the freebsd tree is not up to date.

Cheers
Luigi

On Fri, Sep 16, 2016 at 4:57 PM, Slawa Olhovchenkov  wrote:
> I am wrote NETMAP application and meet some limitation.
> I am use server 16-core server.
> Workers thread run mostly independend, with minimal interworking.
> I am use balancer threads for distributing packet flow between worker
> threads.
> I am using two NETMAP pipes between balancer and workers.
> As result, for case of 4 balancer and 12 workers I am need 4x12x2 = 96
> pipes. NETMAP have limitation of 64 pipes.
>
> May be exist more optimal way?
>
> May be I can emulate tilera way, with bufers credits?
> (In this case balancer give RX buffers, confirm to kernel and pass it
> to worker. kernel reduce credit, used for reciving other buffer from
> pool. After complete worker pass buffer to balancer and balancer
> return buffer to kernel, kernel rise credits and put free buffer to
> pool).
> In this case I am not need to many pipes and not limited performance
> by too long processing of one packet.
> ___
> freebsd-net@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"



-- 
-+---
 Prof. Luigi RIZZO, ri...@iet.unipi.it  . Dip. di Ing. dell'Informazione
 http://www.iet.unipi.it/~luigi/. Universita` di Pisa
 TEL  +39-050-2217533   . via Diotisalvi 2
 Mobile   +39-338-6809875   . 56122 PISA (Italy)
-+---
___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap Checksum Offloading

[Michael Tuexen]

> On 16 Jun 2016, at 08:40, Dominik Schoeffmann  wrote:
> 
> Is the checksum offloading patch for the igb(4) driver available online?
> (I could not find it)
> I would really like to take a look at it, espacially the context
> descriptor part of it.
I would also be interested in it... For looking at SCTP checksum offloading.

Best regards
Michael
> 
> Best regards,
> Dominik
> 
> On 16.06.2016 02:04, Jim Thompson wrote:
>> 
>> Luiz Otavio O Souza (loos@) developed these for igb(4) and, by extension, 
>> em(4) for use in netmap-fwd.
>> 
>> He’s just gone back to Brazil with 82599 ixgb(4) hardware.  I’m sure he’ll 
>> develop similar patches for ixgb(4) in the near future.
>> 
>> Chelsio is also “on the list”, but I figured I’d speak to np@ about it 
>> first.  ;-)  
>> We might do ixl(4) as well.  
>> 
>> Before Luiz retired to Brazil, we discussed upstreaming these to FreeBSD.  
>> We’re committed to make it happen, but I doubt they make 11.
>> 
>> Jim
>> 
>>> On Jun 15, 2016, at 6:50 PM, Navdeep Parhar  wrote:
>>> 
>>> On 06/15/2016 16:15, Andrey Yakovlev wrote:
 ive heard on bsdcan this year that some patches exist to add hwcsum 
 offloading to netmap, hope to see it chelsio at least
>>> 
>>> cxgbe/cxl is a bit sneaky and will let you override netmap (on tx only).
>>> The ncxl interfaces declare themselves capable of checksumming but all
>>> such capabilities are disabled by default.  Just enable txcsum on the
>>> interface and the hardware will do checksum insertion on tx.  No way to
>>> solve the rx part entirely within the driver -- netmap has to be willing
>>> to accept checksum related flags from the driver.
>>> 
>>> Regards,
>>> Navdeep
>>> 
 
 -- 
 ./andy
 
 
 14.06.2016, 12:15, "Dominik Schoeffmann" :
> Dear Netmap Developers,
> 
> during the course of my bachelor's thesis, I modified a packet generator
> called MoonGen [1] in order to utilize netmap.
> One key component was to flexibly offload checksums for different kinds
> of packets (IPv4, UDP, TCP).
> The ixgbe netmap patch was modified [2] in order to construct context
> descriptors and suitable data descriptors. This is implemented in less
> than 250 LoC (including pseudo-header calculations).
> The man page states, that checksum offloading is available via ethtool,
> although a solution inside the netmap API might be a cleaner way for
> applications to actually use these features.
> Attached is a graph showing the performance implication of using
> offloading in the current implementation.
> As can be seen, offloading has only a minor impact.
> When regarding this data (and comparing it to other frameworks), please
> keep in mind, that internally a lot of per-packet effort is needed due
> to the software architecture of the packet generator.
> 
> The question being:
> Would it not make sense to include checksum offloading inside of netmap
> in order to accomodate applications operating on layer 3 and above?
> As these programs need to calculate the checksums in software, it would
> be just as fast to move these calculations to the kernel for NICs
> without checksum offloading support (and the kernel would act as a 
> library).
> The problem which currently is imposed by the fact, that netmap exports
> the complete ring, is that context descriptors disrupt the data
> descriptors, which is unpleasant for the application.
> But you may find the data interesting nevertheless.
> 
> Best Regards,
> Dominik Schoeffmann
> 
> [1] https://github.com/dschoeffm/MoonGen/tree/netmap
> [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading
 ___
 freebsd-net@freebsd.org mailing list
 https://lists.freebsd.org/mailman/listinfo/freebsd-net
 To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
 
>>> 
>>> ___
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>> 
> 

___
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Re: Netmap Checksum Offloading

[Dominik Schoeffmann]

Is the checksum offloading patch for the igb(4) driver available online?
(I could not find it)
I would really like to take a look at it, espacially the context
descriptor part of it.

Best regards,
Dominik

On 16.06.2016 02:04, Jim Thompson wrote:
> 
> Luiz Otavio O Souza (loos@) developed these for igb(4) and, by extension, 
> em(4) for use in netmap-fwd.
> 
> He’s just gone back to Brazil with 82599 ixgb(4) hardware.  I’m sure he’ll 
> develop similar patches for ixgb(4) in the near future.
> 
> Chelsio is also “on the list”, but I figured I’d speak to np@ about it first. 
>  ;-)  
> We might do ixl(4) as well.  
> 
> Before Luiz retired to Brazil, we discussed upstreaming these to FreeBSD.  
> We’re committed to make it happen, but I doubt they make 11.
> 
> Jim
> 
>> On Jun 15, 2016, at 6:50 PM, Navdeep Parhar  wrote:
>>
>> On 06/15/2016 16:15, Andrey Yakovlev wrote:
>>> ive heard on bsdcan this year that some patches exist to add hwcsum 
>>> offloading to netmap, hope to see it chelsio at least
>>
>> cxgbe/cxl is a bit sneaky and will let you override netmap (on tx only).
>> The ncxl interfaces declare themselves capable of checksumming but all
>> such capabilities are disabled by default.  Just enable txcsum on the
>> interface and the hardware will do checksum insertion on tx.  No way to
>> solve the rx part entirely within the driver -- netmap has to be willing
>> to accept checksum related flags from the driver.
>>
>> Regards,
>> Navdeep
>>
>>>
>>> -- 
>>> ./andy
>>>
>>>
>>> 14.06.2016, 12:15, "Dominik Schoeffmann" :
 Dear Netmap Developers,

 during the course of my bachelor's thesis, I modified a packet generator
 called MoonGen [1] in order to utilize netmap.
 One key component was to flexibly offload checksums for different kinds
 of packets (IPv4, UDP, TCP).
 The ixgbe netmap patch was modified [2] in order to construct context
 descriptors and suitable data descriptors. This is implemented in less
 than 250 LoC (including pseudo-header calculations).
 The man page states, that checksum offloading is available via ethtool,
 although a solution inside the netmap API might be a cleaner way for
 applications to actually use these features.
 Attached is a graph showing the performance implication of using
 offloading in the current implementation.
 As can be seen, offloading has only a minor impact.
 When regarding this data (and comparing it to other frameworks), please
 keep in mind, that internally a lot of per-packet effort is needed due
 to the software architecture of the packet generator.

 The question being:
 Would it not make sense to include checksum offloading inside of netmap
 in order to accomodate applications operating on layer 3 and above?
 As these programs need to calculate the checksums in software, it would
 be just as fast to move these calculations to the kernel for NICs
 without checksum offloading support (and the kernel would act as a 
 library).
 The problem which currently is imposed by the fact, that netmap exports
 the complete ring, is that context descriptors disrupt the data
 descriptors, which is unpleasant for the application.
 But you may find the data interesting nevertheless.

 Best Regards,
 Dominik Schoeffmann

 [1] https://github.com/dschoeffm/MoonGen/tree/netmap
 [2] https://github.com/dschoeffm/netmap/tree/mg-chksum-offloading
>>> ___
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>>
>>
>> ___
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
> 



signature.asc
Description: OpenPGP digital signature