It may be that your is not a deadlock but some kind of crash. Enabling debugging features would probably help (e.g. to get a stack trace). Maybe your lockup/crash happened because you did some reconfiguration (ring size, number of rings, etc.) while netmap was active and doing so you triggered some hidden bug.
You probably need to enable all these options https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-options.html 2017-11-21 10:58 GMT+01:00 Harry Schmalzbauer <free...@omnilan.de>: > Bezüglich Vincenzo Maffione's Nachricht vom 21.11.2017 09:39 (localtime): > > Hi, > > It's hard to say, specially because it happens after two days of > > normal use. > > Can't you enable deadlock debugging features in your kernel? > > https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/ > kerneldebug-deadlocks.html > > > > However, if I understand correctly you have created some VLAN interfaces > > vlan0, vlan1, vlan2, ... on top of a NIC (say em0). And you have > > attached each VLAN interface to a vale switch: > > > > # vale-ctl -a vale0:vlan0 > > # vale-ctl -a vale1:vlan1 > > # vale-ctl -a vale2:vlan2 > > > > and each VALE switch is attached to a different set of bhyve guests. > > Hello Vincenzo, > > thank you very much for your help again! > > Your assumption is correct, here's my vale-ctl: > 603.811416 bdg_ctl [148] bridge:0 port:0 vale1:nic1_dmz > 603.811428 bdg_ctl [148] bridge:0 port:1 vale1:styx0 > 603.811430 bdg_ctl [148] bridge:0 port:2 vale1:korso > 603.811432 bdg_ctl [148] bridge:0 port:3 vale1:kallisto > > 603.811434 bdg_ctl [148] bridge:1 port:0 vale2:nic1_inop > 603.811435 bdg_ctl [148] bridge:1 port:1 vale2:styx0 > > 603.811437 bdg_ctl [148] bridge:2 port:0 vale3:nic1_vnl > 603.811439 bdg_ctl [148] bridge:2 port:1 vale3:styx0 > > 603.811441 bdg_ctl [148] bridge:3 port:0 vale4:nic1_egn > 603.811442 bdg_ctl [148] bridge:3 port:1 vale4:styx0 > 603.811444 bdg_ctl [148] bridge:3 port:2 vale4:preed > … > > > > If this is the case, although you are allowed to do that, I don't think > > it's a convenient way to use netmap. > > Since VLAN interfaces like vlan0 do not have (and cannot have) native > > netmap support, you are falling back to emulated netmap adapters (which > > are probably buggy on FreeBSD, specially when combined with VALE). > > Apart from bugs I think that with this setup you can't get decent > > performance that would justify using netmap rather than the standard > > kernel bridge and TAP devices. > > I'm aware about the lost netmap-performace-benefit due to emulated > netmap fallback. > But there were some resonons why I chose vale(4) instead if_bridge(4): > > 1) Inter-Guest-traffic (virtio-net causes lot of LAPIC/IRQ overhead, but > still less overhead than tap(4)/if_bridge(4)) > > 2) Future ptnetmap(4) upgrade path (which should save a lot of LAPIC/IRQ > CPU cycles and unleash huge performace benefits with inter-vm traffic) > > 3) Admin-mess and MTU limitation. Each if_bridge(4) causes a host-stack > interface, which I don't use and which spams ifconfig(8) output; which > if_vtnet(4) even doubles. > Most important disadvantage: if_bridge(4) needs all members to have > exactly the same MTU. This has been a problem for me many times over > the last years in various setups. > > So with my current setup the overhead/efficiency of host-external packet > flow of > bhyve_virtio-net+dyn_vale_port+vale(4) > is equal to > bhyve_virtio-net+if_vtnet(4)+if_bridge(4) > > But I have less disadvanteges with vale(4); as long as emulated netmap > mode doesn't destabilize my setup :-( > > > My second choice was ng_bridge(4). Which I made great experiences in my > router-vm, running on that host in question (and in turn uses virtio-net > interfaces attached to the individual vale(4) switches on the host). > [ Even more impressive: pf(4) runs in a VIMAGE jail in that guest, > utilizing those vale(4) interfaces. Reason for that complicated setup: > Closest hardware abstraction possible. The setup (guest) should be > easily migratable to real hardware ]. > > > > The right way to do it imho would be to write your own (userspace) > > netmap application that forwards packets between your bhyve guests and > > the NIC, prepending/stripping VLAN headers according to configuration > > (e.g. guest A is configured to be on VLAN 100, guest B on VLAN 200), etc. > > I think this would be a very interesting netmap application in general, > > and more importantly you would get the performance that you can't get > > with your setup. > > I agree that having a userland application which, like you described, > utilizes netmap to enable minimalistic SDN features, would be a great > solution. But I would need really a lot of time, since my C skills are > lousy, and I really don't have any time, not even one more day. > I see. But just FYI, there isn't that much to implement :) Cheers, Vincenzo > > > I'll see if I can get any useful information with the kernel deadlock > debuging feature you suggested > (https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/ > kerneldebug-deadlocks.html), > as soon as the problem shows up again. > Since I forgot to add all production-RAM, I had to shutdown yesterday, > so the lockup counter was reset ;-) > Another last-minute change was with netmap ring size: I changed the > vale-uplink interface. The one I used for passthrough had 2 queues > (with EM_MULTIQUEUE support) and the one for the vale uplink onyl one, > and during evaluation phase I reduced rx/tx descriptors to make netmap's > default ring size working. > Now I use the 2-queue NIC with vale uplink and increased ring size to > 81920 while leaving the hardware default of 4096 rx/tx desriptors. > > But my wording wasn't technically correct I think, because I guess what > I'm suffering isn't a real deadlock in terms of locking, but any > netmap-internal lockup/overflow/limit/whatever. Just guesing here! I > don't know netmap code! I only link symptoms, and since that setup is > working really nice for some limited time, I hoped you or any other > netmap expert could teach me how to find the root cause. > Your sentence about FreeBSD's netmap-interface-emulation leaves a bad > presentiment... > > Thank you very much, > > -harry > -- Vincenzo Maffione _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
