Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/25/20 10:24 AM, D'Arcy Cain wrote:

Things got stranger.  First of all, here is my current setup:


And embarrassing but still strange.  Turns out that my problem had nothing 
to do with bridging.  I still don't know what is wrong but I am going to 
continue in the networking list.  Virtualization is not the issue.


For future searches, here is how I set up my system.  In rc.conf I did this:
  hostname="vm01.vex.net"
  me=41 # last octet of IP in hex
  vm_list="" # fill in as you create VMs

I then include a common rc.conf that includes this:
  set -- $(/sbin/ifconfig -l ether); eth0=$1 eth1=$2
  eval "ifconfig_${eth0}_name=\"eth0\""
  eval "ifconfig_${eth1}_name=\"eth1\""
  ifconfig_eth0="inet 0x629e8b${me}/27 up"
  ifconfig_eth1="inet 0xc0a897${me}/24 up"
  ifconfig_eth0_ipv6="inet6 2605:2600:1001::${me}/64 up"
  ifconfig_eth1_ipv6="inet6 fc00:97:97::${me}/64 up"
  vm_enable="YES"
  vm_dir="zfs:zroot/VM"
  vm_delay="5"
  defaultrouter="98.158.139.94"
  ipv6_defaultrouter=2605:2600:1001::1

In rc.local I do this:
  sysctl -w net.inet.ip.forwarding=1 # could be added to sysctl.conf
  sysctl -w net.inet6.ip6.forwarding=1
  vm switch create public
  vm switch add public eth0
  vm switch create private
  vm switch add private eth1

Now VMs are on the local network.

--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/22/20 3:56 AM, Patrick M. Hausen wrote:

Hi!


Am 22.10.2020 um 04:47 schrieb D'Arcy Cain :
public: flags=8843 metric 0 mtu 1500
ether 02:9d:b2:b8:78:00
inet 98.158.139.65 netmask 0xffe0 broadcast 98.158.139.95
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: eth0 flags=143
ifmaxaddr 0 port 1 priority 128 path cost 55
groups: bridge
nd6 options=9
tap0: [...]


tap0 is not a member of your bridge. With the VM running you can try

ifconfig public addm tap0

and check if that changes things.


Things got stranger.  First of all, here is my current setup:

In rc.conf:
  set -- $(/sbin/ifconfig -l ether); eth0=$1 eth1=$2
  eval "ifconfig_${eth0}_name=\"eth0\""
  eval "ifconfig_${eth1}_name=\"eth1\""
  ifconfig_eth0="-tso -lro -rxcsum -rxcsum6 -txcsum -txcsum6 -vlanhwtag 
-vlanhwtso up"


In rc.local:
  vm switch create public
  vm switch add public eth0
  ifconfig vm-public inet 0x629e8b41/27
  ifconfig vm-public inet6 2605:2600:1001::41/64
  route add default 98.158.139.94
  route add -inet6 default 2605:2600:1001::1

This works just like a number of previous attempts.  I can ping any site in 
the world but, other than to the host, I can't connect by TCP.  My test is 
ssh.  But now I just noticed that I can ssh into one other server on my 
network.  The only difference that I can find is that the one that I can 
connect to has a HP NC382i DP Multifunction Gigabit Server Adapter - bce(5). 
 It's the only one in the network with that adapter.  The host and many 
other servers are bge(5).  Here are the ifconfig entries:


eth0: flags=8943 metric 0 
mtu 1500

options=80088
ether 14:02:ec:31:60:d0
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29
vm-public: flags=8843 metric 0 mtu 1500
ether 1a:d8:8b:3e:51:87
inet 98.158.139.65 netmask 0xffe0 broadcast 98.158.139.95
inet6 2605:2600:1001::41 prefixlen 64
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143
ifmaxaddr 0 port 8 priority 128 path cost 200
member: eth0 flags=143
ifmaxaddr 0 port 1 priority 128 path cost 2
groups: bridge vm-switch viid-4c918@
nd6 options=1

And the remote:
bce0: flags=8943 metric 0 
mtu 1500

options=c00b9
ether 3c:d9:2b:f9:e2:10
inet 98.158.139.77 netmask 0xffe0 broadcast 98.158.139.95
inet6 fe80::3ed9:2bff:fef9:e210%bce0 prefixlen 64 scopeid 0x1
inet6 2605:2600:1001::4d prefixlen 64
media: Ethernet autoselect (100baseTX )
status: active
nd6 options=21

That PROMISC flag looked promising but turning it on on other servers didn't 
help.  In any case I hope I can fix this on my host.  I think that I may 
find it hard to have everyone else in the world change their system.


--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[Jason Tubnor]

On Fri, 23 Oct 2020 at 23:46, D'Arcy Cain  wrote:

>
>
> Actually, I can make a TCP connection from the VM to the host.  Do I have
> to
> set up some sort of proxy arp?  How would I do that?
>
>
>
In 12.1/2, we experienced issues similar, turning off LRO (-lro) on the
physical interface fixed this for us.  LRO causes packet fragmentation and
I am sure you'd get ICMP to fail if you increased the payload.

Cheers,

Jason.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/23/20 12:11 AM, Jason Tubnor wrote:
If both your hosts are configured using the same naming conventions for 
bridges and vm-bhyve switches, migration should be pretty simple and painless.


I had to modify vm-bhyve to get it to use my switch names so that works now. 
 It still didn't add it automatically so I had to run "ifconfig public addm 
tap0" manually.


As a result I am right back to where I was.  I can ping any IP address on 
the net but I cannot make a TCP connection.  I can't even use domain names 
because I can't connect to the DNS server in my own network.


Actually, I can make a TCP connection from the VM to the host.  Do I have to 
set up some sort of proxy arp?  How would I do that?


--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/22/20 6:50 PM, Paul Vixie wrote:

here's how i do it, per years-ago documentation to this effect:

autobridge_interfaces="bridge0" autobridge_bridge0="tap* igb1" 


Will that work if I have two networks?

autobridge_interfaces="bridge0" autobridge_bridge0="tap* bge0"
autobridge_interfaces="bridge1" autobridge_bridge1="tap* bge1"

Seems like it would get confused about which tap to assign to which bridge.

ifconfig_igb1="up media 1000baseTX fib 1" cloned_interfaces="bridge0 tap0 
tap1 tap2 tap3 tap4 tap5 tap6 tap7"


Again, they aren't all assigned to one bridge.  Probably the even taps would 
go to one and the odd taps to the other but that's not guaranteed.


ifconfig_bridge0="inet 24.104.150.210/27 fib 1" 
ifconfig_bridge0_ipv6="inet6 2001:559:8000:cd::2/64 fib 1 auto_linklocal up"

ifconfig_tap0="up fib 1"
ifconfig_tap1="up fib 1"
ifconfig_tap2="up fib 1"
ifconfig_tap3="up fib 1"
ifconfig_tap4="up fib 1"
ifconfig_tap5="up fib 1"
ifconfig_tap6="up fib 1"
ifconfig_tap7="up fib 1"


note, fib 1 is a detail here, just gives me a different default route for 
the virtual machines.


In my case I want to make the default route the same as the host's.


autobridge_* (rc.conf(5)) is what you'll need for that.


See above.

--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[Jason Tubnor]

On Fri, 23 Oct 2020 at 01:03, D'Arcy Cain  wrote:

>
> >
> >
> > tap0 is not a member of public so has no way to get to eth0 or public.
>
> Odd.  I thought that vm-bhyve did that when the VM came up.
>
> Hmm.  It looks like tap0 gets attached to vm-public if it is configured to
> attach to public.  Perhaps I just need to name the bridge that way.
>

If you create the virtual switch (bridge) for vm-bhyve using:

vm switch create -b  

Then for the guest:

vm add -d network -s  

When you start your guest up, it will attach to the existing bridge you
defined in rc.conf, each time.


>
> > vm add -d network -s public 
>
> Hard to automate when VM can migrate to another host.
>
>
If both your hosts are configured using the same naming conventions for
bridges and vm-bhyve switches, migration should be pretty simple and
painless.

Cheers,

Jason.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: When is a switch not a switch?

[Paul Vixie]

D'Arcy Cain wrote on 2020-10-22 07:03:
> On 10/21/20 11:16 PM, Jason Tubnor wrote:
>>     public: flags=8843 metric
>> 0 mtu 1500
>>   ...
>>           groups: bridge
>>           nd6 options=9
>>     tap0: flags=8843 metric 0
>> mtu 1500
>> ...
>>           groups: tap vm-port
>>           media: Ethernet autoselect
>>           ...
>>
>>
>> tap0 is not a member of public so has no way to get to eth0 or public.
>
> Odd.  I thought that vm-bhyve did that when the VM came up.

here's how i do it, per years-ago documentation to this effect:

> autobridge_interfaces="bridge0"   
>  
> autobridge_bridge0="tap*
> igb1" 
> 
> ifconfig_igb1="up media 1000baseTX fib
> 1"
>   
> cloned_interfaces="bridge0 tap0 tap1 tap2 tap3 tap4 tap5 tap6 tap7"
> ifconfig_bridge0="inet 24.104.150.210/27 fib
> 1"
> ifconfig_bridge0_ipv6="inet6 2001:559:8000:cd::2/64 fib 1
> auto_linklocal up"
> ifconfig_tap0="up fib 1"
> ifconfig_tap1="up fib 1"
> ifconfig_tap2="up fib 1"
> ifconfig_tap3="up fib 1"
> ifconfig_tap4="up fib 1"
> ifconfig_tap5="up fib 1"
> ifconfig_tap6="up fib 1"
> ifconfig_tap7="up fib 1"

note, fib 1 is a detail here, just gives me a different default route
for the virtual machines.


>
> Hmm.  It looks like tap0 gets attached to vm-public if it is
> configured to attach to public.  Perhaps I just need to name the
> bridge that way.

autobridge_* (rc.conf(5)) is what you'll need for that.

>
>> I'd avoid creating the 'public' bridge and let vm-bhyve create it. 
>> Something like:
>>
>> vm switch create public
>> vm switch add public eth0
>
> But then I can't create the IP on the host until too late in the
> boot.  I assume that you mean to do that in rc.local.  Are you sure
> that the above is any different than what I did?

because my management interface for the milking machine is on a
different subnet, i have this:

> vlans_igb0="201 203"
> ifconfig_igb0="up vlanmtu media
> 1000baseTX"   
>  
> ifconfig_igb0_201="inet
> 24.104.150.130/26"
>  
> ifconfig_igb0_201_ipv6="inet6 2001:559:8000:c9::3/64"
> ifconfig_igb0_203="inet
> 24.104.150.16/27" 
>  
> ifconfig_igb0_203_ipv6="inet6 2001:559:8000:cb::16/64"


however, you could do away with that and just autobridge your connected
interface (mine is igb1, see above.)

>
>> Then for the guest:
>>
>> vm add -d network -s public 
>
> Hard to automate when VM can migrate to another host.

hard to do any of this if you're not doing it by hand. (i don't use the
"vm" command.)

-- 
Sent from Postbox

___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/21/20 11:16 PM, Jason Tubnor wrote:

public: flags=8843 metric 0 mtu 1500
          ether 02:9d:b2:b8:78:00
          inet 98.158.139.65 netmask 0xffe0 broadcast 98.158.139.95
          id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
          maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
          root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
          member: eth0 flags=143
                  ifmaxaddr 0 port 1 priority 128 path cost 55
          groups: bridge
          nd6 options=9
tap0: flags=8843 metric 0 mtu 1500
          description: vmnet-BASE-0-public
          options=8
          ether 00:bd:5f:56:f8:00
          groups: tap vm-port
          media: Ethernet autoselect
          status: active
          nd6 options=29
          Opened by PID 3679


tap0 is not a member of public so has no way to get to eth0 or public.


Odd.  I thought that vm-bhyve did that when the VM came up.

Hmm.  It looks like tap0 gets attached to vm-public if it is configured to 
attach to public.  Perhaps I just need to name the bridge that way.


I'd avoid creating the 'public' bridge and let vm-bhyve create it.  
Something like:


vm switch create public
vm switch add public eth0


But then I can't create the IP on the host until too late in the boot.  I 
assume that you mean to do that in rc.local.  Are you sure that the above is 
any different than what I did?



Then for the guest:

vm add -d network -s public 


Hard to automate when VM can migrate to another host.

--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[Patrick M. Hausen]

Hi!

> Am 22.10.2020 um 04:47 schrieb D'Arcy Cain :
> public: flags=8843 metric 0 mtu 1500
>ether 02:9d:b2:b8:78:00
>inet 98.158.139.65 netmask 0xffe0 broadcast 98.158.139.95
>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>member: eth0 flags=143
>ifmaxaddr 0 port 1 priority 128 path cost 55
>groups: bridge
>nd6 options=9
> tap0: [...]

tap0 is not a member of your bridge. With the VM running you can try

ifconfig public addm tap0

and check if that changes things.

Then go back to the drawing board and probably let vm-bhyve manage
all that stuff. Just make sure to configure your physical interface with
the hardware acceleration features disabled.

You *can* put the IP address on the physical interface and have
vm-bhyve create the bridge. I honestly don't know why the documentation
explicitly states that you should not. FreeNAS has been running like this
for years and only supports the "correct" configuration since 11.3 or so.

On the other hand coming from Cisco and friends putting the address on
the topmost layer 3 interface does make perfect sense to me - so e.g.
on a Cisco switch you have physical ports that are members of a VLAN
and if you run anything layer 3 on that box, of course the address goes
on the VLAN, not the port ...

But give vm-bhyve a spin with the address on the physical. Or use two
different physical interfaces - one for the host, one for the "public" bridge.

HTH,
Patrick
--
punkt.de GmbH
Patrick M. Hausen
.infrastructure

Kaiserallee 13a
76133 Karlsruhe

Tel. +49 721 9109500

https://infrastructure.punkt.de
i...@punkt.de

AG Mannheim 108285
Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein



signature.asc
Description: Message signed with OpenPGP

Re: When is a switch not a switch?

[Jason Tubnor]

On Thu, 22 Oct 2020 at 13:47, D'Arcy Cain  wrote:

>
> Here is what the interfaces look like:
>
> eth0: flags=8943 metric 0
> mtu 1500
>  options=80088
>  ether 14:02:ec:31:60:d0
>  media: Ethernet autoselect (1000baseT )
>  status: active
>  nd6 options=29
> public: flags=8843 metric 0 mtu
> 1500
>  ether 02:9d:b2:b8:78:00
>  inet 98.158.139.65 netmask 0xffe0 broadcast 98.158.139.95
>  id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>  maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>  root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>  member: eth0 flags=143
>  ifmaxaddr 0 port 1 priority 128 path cost 55
>  groups: bridge
>  nd6 options=9
> tap0: flags=8843 metric 0 mtu 1500
>  description: vmnet-BASE-0-public
>  options=8
>  ether 00:bd:5f:56:f8:00
>  groups: tap vm-port
>  media: Ethernet autoselect
>  status: active
>  nd6 options=29
>  Opened by PID 3679
>

tap0 is not a member of public so has no way to get to eth0 or public.

I'd avoid creating the 'public' bridge and let vm-bhyve create it.
Something like:

vm switch create public
vm switch add public eth0

Then for the guest:

vm add -d network -s public 

If you have IPv4/v6 addresses on eth0, you will be able to ping those from
the guest.

Cheers,

Jason.

In the VM I set the IP to 71 on the same network.  Here is what that looks
> like:
>
> vtnet0: flags=8943 metric
> 0
> mtu 1500
>  options=80028
>  ether 22:22:22:22:22:00
>  inet 98.158.139.71 netmask 0xffe0 broadcast 98.158.139.95
>  media: Ethernet 10Gbase-T 
>  status: active
>  nd6 options=29
>
> Everything looks correct but I can't even ping between the host and the VM.
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/20/20 7:52 AM, Patrick M. Hausen wrote:

What you need to configure depends on your hardware.
I made a table for the various interfaces we use at our place:

em: -rxcsum -txcsum -lro -vlanmtu -vlanhwcsum -vlanhwfilter 
-vlanhwtag up
igb:-rxcsum -rxcsum6 -txcsum -txcsum6 -tso -vlanhwtag -vlanhwtso up


I have bge(4) cards.  This seems to be the correct one for me.

It still doesn't work.  Here is the relevant lines from my rc.conf.  I may 
as well use the actual values.  This is verbatim.


  ifconfig_eth0="-rxcsum -rxcsum6 -txcsum -txcsum6 -tso -vlanhwtag 
-vlanhwtso up"

  cloned_interfaces="bridge0 bridge1"
  ifconfig_bridge0_name="public"
  ifconfig_public="addm eth0 up"
  ifconfig_public_alias0="inet 0x629e8b${me}/27"
  ifconfig_public_alias0_ipv6="inet6 2605:2600:1001::${me}/64"
  defaultrouter="98.158.139.94"
  ipv6_defaultrouter=2605:2600:1001::1

Note that "me" is set to the hex value of the last octet, 65 in this case.

Here is what the interfaces look like:

eth0: flags=8943 metric 0 
mtu 1500

options=80088
ether 14:02:ec:31:60:d0
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29
public: flags=8843 metric 0 mtu 1500
ether 02:9d:b2:b8:78:00
inet 98.158.139.65 netmask 0xffe0 broadcast 98.158.139.95
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: eth0 flags=143
ifmaxaddr 0 port 1 priority 128 path cost 55
groups: bridge
nd6 options=9
tap0: flags=8843 metric 0 mtu 1500
description: vmnet-BASE-0-public
options=8
ether 00:bd:5f:56:f8:00
groups: tap vm-port
media: Ethernet autoselect
status: active
nd6 options=29
Opened by PID 3679

In the VM I set the IP to 71 on the same network.  Here is what that looks like:

vtnet0: flags=8943 metric 0 
mtu 1500

options=80028
ether 22:22:22:22:22:00
inet 98.158.139.71 netmask 0xffe0 broadcast 98.158.139.95
media: Ethernet 10Gbase-T 
status: active
nd6 options=29

Everything looks correct but I can't even ping between the host and the VM.

--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[John-Mark Gurney]

D'Arcy Cain wrote this message on Tue, Oct 20, 2020 at 05:28 -0400:
> On 10/20/20 4:36 AM, Patrick M. Hausen wrote:
> > It's officially documented here:
> > https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-bridging.html
> 
> I did see that.  Does that mean that I don't even need to create switches at 
> all?
> 
> > "If the bridge host needs an IP address, set it on the bridge interface, 
> > not on the member interfaces."
> 
> But I don't necessarily need an IP on the bridge itself, right?

I can't say w/o more info in exactly how your system is setup...  If
you could provide more concrete information about what interfaces you
are using, and what IPs are configured on what interfaces (or VMs),
that'd be helpful.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."


signature.asc
Description: PGP signature

Re: When is a switch not a switch?

[Jason Tubnor]

Hi,

On Tue, 20 Oct 2020 at 13:02, D'Arcy Cain  wrote:

> I am using bhyve with vm-bhyve,  I am trying to set up a virtual network
> with multiple hosts.  The idea is that a VM would be on the same virtual
> network no matter which actual host it is on.
>
> Say I have a public network a.b.c.0/24.  I thought I could create a switch
> on a host.  The host would be a.b.c.1 and the VMs would be a.b.c.100 and
> a.b.c.101.  The idea would be that the VMs would appear on the real
> network.
>   Then the 101 VM could migrate to a.b.c.2 and still be accessible.  I
> envisioned some sort of proxy arp would happen so that every VM would
> simply
> announce itself wherever it was.
>

It looks like you are over complicating this.  When using vm-bhyve, as long
as each host has the same vswitch (bridge) then the tap will automagically
be inserted correctly on guest startup (as long as the conf file follows
your guest storage). Let vm-bhyve manage bridge creation.

Only use /etc/rc.conf to bring up the interface.  If you are running > 11.4
then you must turn LRO off (-lro) when you bring up the interface.  The
other settings in this thread can be left on.  The problem you are
experiencing is packet fragmentation that the guest has to deal with
because LRO is enabled (off by default in 11, enabled in 12 and above).
LRO should be disabled automatically when an interface (or child VLAN) is
added to a bridge.  I have tried to get the network guys to fix this but no
such luck.


> This did seem to work in that I could ping from the VM:
>
> # ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> 64 bytes from 8.8.8.8: icmp_seq=0 ttl=114 time=1.734 ms
>
> Even IPV6:
>
> # ping6 2605:2600:1001::4b
> PING6(56=40+8+8 bytes) 2605:2600:1001::4 --> 2605:2600:1001::4b
> 16 bytes from 2605:2600:1001::4b, icmp_seq=0 hlim=64 time=0.960 ms
> 16 bytes from 2605:2600:1001::4b, icmp_seq=1 hlim=64 time=0.415 ms
>
> However TCP doesn't work.  In fact, I could only ping by IP because the
> system couldn't connect to the DNS server, to get an address even though
> it
> could ping it.
>
> I guess my first question is does this seem doable?  If so, what am I
> missing?  Is it possible that a bhyve switch is more like a router?
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/20/20 8:35 AM, Patrick M. Hausen wrote:

So why alias?  Wouldn't "ifconfig_public=" work?


We already have

ifconfig_public="addm bge0 up"

Adding

ifconfig_public="inet 1.2.3.4/24"

on another line would overwrite the first one. These are just
variable assignments not executable code. You cannot have
more than one


Doh!  Of course.  I would have known that after the next coffee.

--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[Shawn Webb]

On Mon, Oct 19, 2020 at 10:02:17PM -0400, D'Arcy Cain wrote:
> I am using bhyve with vm-bhyve,  I am trying to set up a virtual network
> with multiple hosts.  The idea is that a VM would be on the same virtual
> network no matter which actual host it is on.
> 
> Say I have a public network a.b.c.0/24.  I thought I could create a switch
> on a host.  The host would be a.b.c.1 and the VMs would be a.b.c.100 and
> a.b.c.101.  The idea would be that the VMs would appear on the real network.
> Then the 101 VM could migrate to a.b.c.2 and still be accessible.  I
> envisioned some sort of proxy arp would happen so that every VM would simply
> announce itself wherever it was.
> 
> This did seem to work in that I could ping from the VM:
> 
> # ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> 64 bytes from 8.8.8.8: icmp_seq=0 ttl=114 time=1.734 ms
> 
> Even IPV6:
> 
> # ping6 2605:2600:1001::4b
> PING6(56=40+8+8 bytes) 2605:2600:1001::4 --> 2605:2600:1001::4b
> 16 bytes from 2605:2600:1001::4b, icmp_seq=0 hlim=64 time=0.960 ms
> 16 bytes from 2605:2600:1001::4b, icmp_seq=1 hlim=64 time=0.415 ms
> 
> However TCP doesn't work.  In fact, I could only ping by IP because the
> system couldn't connect to the DNS server, to get an address even though it
> could ping it.
> 
> I guess my first question is does this seem doable?  If so, what am I
> missing?  Is it possible that a bhyve switch is more like a router?
> 
> Thanks.
> 
> -- 
> D'Arcy J.M. Cain  |  Democracy is three wolves
> http://www.druid.net/darcy/|  and a sheep voting on
> +1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
> IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net
> 
> Disclaimer: By sending an email to ANY of my addresses you
> are agreeing that:
> 
> 1.  I am by definition, "the intended recipient".
> 2.  All information in the email is mine to do with as I see
> fit and make such financial profit, political mileage, or
> good joke as it lends itself to. In particular, I may quote
> it where I please.
> 3.  I may take the contents as representing the views of
> your company if I so wish.
> 4.  This overrides any disclaimer or statement of
> confidentiality that may be included or implied in
> your message.

I usually configure my bridgeN device to have an IP and subnet that I
know won't be on any of the physical networks I care about. I'll then
add only the tapN..M devices that the bhyve VMs will use to that
bridgeN. I'll then use pf to NAT from that private network on bridgeN
to the real world.

 BEGIN rc.conf 
cloned_interfaces="bridge0 tap0 tap1"

ifconfig_bridge0="inet 192.168.254.1 subnet mask 255.255.255.0"
ifconfig_bridge0="${ifconfig_bridge0} addm tap0 addm tap1"
 END rc.conf 

 BEGIN pf.conf 
table  counters { \
192.168.254.0/24 \
}

scrub in all

nat on em0 from {} to any -> (em0)
nat on wlan0 from {} to any -> (wlan0)

pass in all
pass out all
 END pf.conf 

Thanks,

-- 
Shawn Webb
Cofounder / Security Engineer
HardenedBSD

GPG Key ID:  0xFF2E67A277F8E1FA
GPG Key Fingerprint: D206 BB45 15E0 9C49 0CF9  3633 C85B 0AF8 AB23 0FB2
https://git-01.md.hardenedbsd.org/HardenedBSD/pubkeys/src/branch/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc


signature.asc
Description: PGP signature

Re: When is a switch not a switch?

[Patrick M. Hausen]

Hi!

> Am 20.10.2020 um 14:10 schrieb D'Arcy Cain :
> 
> On 10/20/20 7:39 AM, Patrick M. Hausen wrote:
>>> When I started I thought of a switch as analogous to a physical switch.  If 
>>> I am in an office with one ethernet jack but I have multiple devices I 
>>> might connect a switch (or hub) to the jack and plug my devices into the 
>>> switch. I don't need to create a separate network for my office.  All of my 
>>> devices are on the company network.
>> OK, the "switch" interface in FreeBSD is bridge(4).
> 
> Understood.
> 
>> Or to cite Radia Perlman:
>> A bridge is a network device making forwarding decisions based on layer 2 
>> addresses.
>> A router is a network device making forwarding decisions based on layer 3 
>> addresses.
>> "Switch" is a marketing term meaning "faster or cheaper than the 
>> competition".
> 
> I always thought that a switch was a hub with packet switching to avoid 
> collisions.

That is a bridge. A switch simply is a multiport bridge. And a layer 3 switch 
is a router.

> Or else rename the bridges to "public" and "private".

Yep, probably.

>> ifconfig_inet0="addm igb0 up"
> ifconfig_public="addm bge0 up"
> ifconfig_private="addm bge1 up"
> 
>> ifconfig_inet0_alias0="inet 1.2.3.4/24"
> ifconfig_public_alias0="inet 1.2.3.4/24"
> ifconfig_private_alias0="192.168.151.4/14"
> 
> So why alias?  Wouldn't "ifconfig_public=" work?

We already have

ifconfig_public="addm bge0 up"

Adding

ifconfig_public="inet 1.2.3.4/24"

on another line would overwrite the first one. These are just
variable assignments not executable code. You cannot have
more than one

ifconfig_public

line. If you need more than one they have to be named

ifconfig_public
ifconfig_public_alias0
ifconfig_public_alias1
...

Execution stops at the first undefined one, so no gaps, either.

> Not sure I need this as long as arp works as it should.  Do I really care 
> what the MAC is?

Well, the ARP timeouts specifically of Cisco gear can be enervatingly long so 
hosts
are not reachable after reboot for minutes ... these settings fix that.

HTH,
Patrick
--
punkt.de GmbH
Patrick M. Hausen
.infrastructure

Kaiserallee 13a
76133 Karlsruhe

Tel. +49 721 9109500

https://infrastructure.punkt.de
i...@punkt.de

AG Mannheim 108285
Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein



signature.asc
Description: Message signed with OpenPGP

Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/20/20 7:39 AM, Patrick M. Hausen wrote:

When I started I thought of a switch as analogous to a physical switch.  If I 
am in an office with one ethernet jack but I have multiple devices I might 
connect a switch (or hub) to the jack and plug my devices into the switch. I 
don't need to create a separate network for my office.  All of my devices are 
on the company network.


OK, the "switch" interface in FreeBSD is bridge(4).


Understood.


Or to cite Radia Perlman:

A bridge is a network device making forwarding decisions based on layer 2 
addresses.
A router is a network device making forwarding decisions based on layer 3 
addresses.
"Switch" is a marketing term meaning "faster or cheaper than the competition".


I always thought that a switch was a hub with packet switching to avoid 
collisions.



cloned_interfaces="bridge0"
ifconfig_bridge0="a.b.c.d.1 addm bge0 addm switch0 up"



Except that switch0 doesn't get created until vm-bhyve starts so it probably 
doesn't exist at that time.


What is "switch0"? I suspect it is just a bridge interface that gets renamed by
your VM management software. In that case manually creating bridge0
and all the things we discussed will not get you anywhere.


So in vm-bhyve I need to change;

@@ -3,9 +3,9 @@
 cpu=2
 memory=2G
 network0_type="virtio-net"
-network0_switch="public"
+network0_switch="bridge0"
 network1_type="virtio-net"
-network1_switch="private"
+network1_switch="bridge1"
 disk0_type="virtio-blk"
 disk0_name="disk0.img"
 disk0_dev="sparse-zvol"

Or else rename the bridges to "public" and "private".


Real life example from our environment:

ifconfig_igb0="-rxcsum -rxcsum6 -txcsum -txcsum6 -tso -vlanhwtag -vlanhwtso up"

ifconfig_bge0="-rxcsum -rxcsum6 -txcsum -txcsum6 -tso -vlanhwtag -vlanhwtso up"


cloned_interfaces="bridge0"

cloned_interfaces="bridge0 bridge1"


ifconfig_bridge0_name="inet0"

ifconfig _bridge0_name="public"
ifconfig _bridge1_name="private"


ifconfig_inet0="addm igb0 up"

ifconfig_public="addm bge0 up"
ifconfig_private="addm bge1 up"


ifconfig_inet0_alias0="inet 1.2.3.4/24"

ifconfig_public_alias0="inet 1.2.3.4/24"
ifconfig_private_alias0="192.168.151.4/14"

So why alias?  Wouldn't "ifconfig_public=" work?


Then we configure iocage to attach the jails to bridge0.

In your case you would have to tell your VM management tool to attach the
VM tap interfaces to bridge0 instead of creating its own "switch0" - which I
suspect is a bridge interface in disguise. As you can see above we rename
all our Internet facing interfaces to "inet0" on all hosts. Then there are more
like "mgmt0", "priv0", ... like that. So probably the bridge is renamed to 
"switch0".


If I do the above I guess I can keep the names "public" and "private".



Tell the tool not to do that and use the preconfigured bridge0 instead.


Or public?  inet0 in your example?



Another useful sysctl to get reproduceable static MAC addresses for the bridge
itself accross reboots is:

loader.conf: if_bridge_load="YES"
sysctl.conf: net.link.bridge.inherit_mac=1


Not sure I need this as long as arp works as it should.  Do I really care 
what the MAC is?


Cheers.

--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[Patrick M. Hausen]

A short P.S.

I just looked shortly into vm-bhyve. Whatever this tool does with respect to 
the "switches",
possibly just turning off all the hardware acceleration features for your 
network card can
solve your problems without further messing around with bridge(4) and friends.

What you need to configure depends on your hardware.
I made a table for the various interfaces we use at our place:

em: -rxcsum -txcsum -lro -vlanmtu -vlanhwcsum -vlanhwfilter 
-vlanhwtag up
igb:-rxcsum -rxcsum6 -txcsum -txcsum6 -tso -vlanhwtag -vlanhwtso up
ix: -rxcsum -rxcsum6 -txcsum -txcsum6 -tso -lro -vlanhwtag 
-vlanhwtso up
ixl:-rxcsum -rxcsum6 -txcsum -txcsum6 -tso -lro -vlanhwtag 
-vlanhwtso up
bnxt:   -rxcsum -rxcsum6 -txcsum -txcsum6 -tso -lro -vlanhwtag -vlanhwtso 
-vlanhwfilter up

HTH,
Patrick
--
punkt.de GmbH
Patrick M. Hausen
.infrastructure

Kaiserallee 13a
76133 Karlsruhe

Tel. +49 721 9109500

https://infrastructure.punkt.de
i...@punkt.de

AG Mannheim 108285
Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein



signature.asc
Description: Message signed with OpenPGP

Re: When is a switch not a switch?

[Patrick M. Hausen]

Hi all,

> Am 20.10.2020 um 12:50 schrieb D'Arcy Cain :
> 
> On 10/20/20 5:36 AM, Patrick M. Hausen wrote:
>>> I did see that.  Does that mean that I don't even need to create switches 
>>> at all?
>> What is a switch in this context? I use bridge interfaces to connect jails 
>> via epair
>> and VMs via tap.
> 
> When I started I thought of a switch as analogous to a physical switch.  If I 
> am in an office with one ethernet jack but I have multiple devices I might 
> connect a switch (or hub) to the jack and plug my devices into the switch. I 
> don't need to create a separate network for my office.  All of my devices are 
> on the company network.

OK, the "switch" interface in FreeBSD is bridge(4).

Or to cite Radia Perlman:

A bridge is a network device making forwarding decisions based on layer 2 
addresses.
A router is a network device making forwarding decisions based on layer 3 
addresses.
"Switch" is a marketing term meaning "faster or cheaper than the competition".

> cloned_interfaces="bridge0"
> ifconfig_bridge0="a.b.c.d.1 addm bge0 addm switch0 up"

> Except that switch0 doesn't get created until vm-bhyve starts so it probably 
> doesn't exist at that time.

What is "switch0"? I suspect it is just a bridge interface that gets renamed by
your VM management software. In that case manually creating bridge0
and all the things we discussed will not get you anywhere.

>> If em0 does not have an IP address on the host and should be used
>> exclusively for VMs, then the bridge does not need an IP address, either.
>> Still you need to configure em0 "up".
> 
> I can't imagine a scenario like that.  You probably always need access to the 
> host for maintenance.

Well, there could be a second hardware interface for host communication ...
And if one of the two is member of the bridge and the other one isn't it is
perfectly valid to plug them into the same broadcast domain and get
e.g. 1Gbit/s for the host and 1Gbit/s for all the jails or VMs.

>> And additionally ...
>> - you should disable all hardware acceleration features on the physical 
>> interface
> 
> Like ASF?

Real life example from our environment:

ifconfig_igb0="-rxcsum -rxcsum6 -txcsum -txcsum6 -tso -vlanhwtag -vlanhwtso up"
cloned_interfaces="bridge0"
ifconfig_bridge0_name="inet0"
ifconfig_inet0="addm igb0 up"
ifconfig_inet0_alias0="inet 1.2.3.4/24"

Then we configure iocage to attach the jails to bridge0.

In your case you would have to tell your VM management tool to attach the
VM tap interfaces to bridge0 instead of creating its own "switch0" - which I
suspect is a bridge interface in disguise. As you can see above we rename
all our Internet facing interfaces to "inet0" on all hosts. Then there are more
like "mgmt0", "priv0", ... like that. So probably the bridge is renamed to 
"switch0".

Tell the tool not to do that and use the preconfigured bridge0 instead.

Another useful sysctl to get reproduceable static MAC addresses for the bridge
itself accross reboots is:

loader.conf: if_bridge_load="YES"
sysctl.conf: net.link.bridge.inherit_mac=1


HTH,
Patrick
--
punkt.de GmbH
Patrick M. Hausen
.infrastructure

Kaiserallee 13a
76133 Karlsruhe

Tel. +49 721 9109500

https://infrastructure.punkt.de
i...@punkt.de

AG Mannheim 108285
Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein


signature.asc
Description: Message signed with OpenPGP

Re: When is a switch not a switch?

[Patrick M. Hausen]

Hi all,

> Am 20.10.2020 um 11:28 schrieb D'Arcy Cain :
> 
> On 10/20/20 4:36 AM, Patrick M. Hausen wrote:
>> It's officially documented here:
>> https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-bridging.html
> 
> I did see that.  Does that mean that I don't even need to create switches at 
> all?

What is a switch in this context? I use bridge interfaces to connect jails via 
epair
and VMs via tap.

>> "If the bridge host needs an IP address, set it on the bridge interface, not 
>> on the member interfaces."
> 
> But I don't necessarily need an IP on the bridge itself, right?

Depends ;-)

If the host has got e.g. em0 with an IP address and you want to make
that physical interface part of e.g. bridge0 as well as all the VMs so they
can communicate on the wire ...

you *must* move the IP address config from em0 to bridge0 and
configure em0 "up".

If em0 does not have an IP address on the host and should be used
exclusively for VMs, then the bridge does not need an IP address, either.
Still you need to configure em0 "up".

And additionally ...

- you should disable all hardware acceleration features on the physical 
interface
- if you are using pf you should move the rule processing from the members to 
the bridge like so:

sysctl net.link.bridge.pfil_member=0
sysctl net.link.bridge.pfil_bridge=1


HTH,
Patrick
--
punkt.de GmbH
Patrick M. Hausen
.infrastructure

Kaiserallee 13a
76133 Karlsruhe

Tel. +49 721 9109500

https://infrastructure.punkt.de
i...@punkt.de

AG Mannheim 108285
Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein


signature.asc
Description: Message signed with OpenPGP

Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/20/20 4:36 AM, Patrick M. Hausen wrote:

It's officially documented here:
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-bridging.html


I did see that.  Does that mean that I don't even need to create switches at 
all?



"If the bridge host needs an IP address, set it on the bridge interface, not on the 
member interfaces."


But I don't necessarily need an IP on the bridge itself, right?

--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[D'Arcy Cain]

On 10/20/20 2:56 AM, John-Mark Gurney wrote:

By switch, do you mean use bridge?  How specifically is the network
configured?


Yes.  I did try bridge first but I may not have understand all the nuances. 
 I also thought that "switch" meant the same thing as a physical switch but 
I guess it is more like a router.  That was the point of my subject.



What you are describing sounds like what I do w/ bridge, but my use was
slightly more complicated.

Say your host has em0 as the main network, you would create a bridge0
interface, either via cloned_interfaces or via "ifconfig bridge0 create".
Then you would put the em0 interface as a member of the bridge


"ifconfig bridge0 addm bge0" in my case but I also have a private network so 
"ifconfig bridge0 addm bge0 addm bge1" then.  Or do I need two bridges?



interface.  You would also add the tap interfaces of the various bhyve
vms as well (don't forget to make sure the tap interface is up on the
host, net.link.tap.up_on_open helps w/ this)...


This is the part I am trying to automate so that VM can freely move between 
hosts.  Is there a way to make tap automatically add itself to a bridge?


Thanks for your help.

--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature

Re: When is a switch not a switch?

[Patrick M. Hausen]

Hi all,

> Am 20.10.2020 um 08:56 schrieb John-Mark Gurney :
> I have heard (and that is the way I do that), that you have to put the
> host IPs on the bridge0 interface, and not the em0 interface.

It's officially documented here:
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/network-bridging.html

"If the bridge host needs an IP address, set it on the bridge interface, not on 
the member interfaces."


Kind regards,
Patrick
--
punkt.de GmbH
Patrick M. Hausen
.infrastructure

Kaiserallee 13a
76133 Karlsruhe

Tel. +49 721 9109500

https://infrastructure.punkt.de
i...@punkt.de

AG Mannheim 108285
Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein


signature.asc
Description: Message signed with OpenPGP

Re: When is a switch not a switch?

[John-Mark Gurney]

D'Arcy Cain wrote this message on Mon, Oct 19, 2020 at 22:02 -0400:
> I am using bhyve with vm-bhyve,  I am trying to set up a virtual network 
> with multiple hosts.  The idea is that a VM would be on the same virtual 
> network no matter which actual host it is on.
> 
> Say I have a public network a.b.c.0/24.  I thought I could create a switch 
> on a host.  The host would be a.b.c.1 and the VMs would be a.b.c.100 and 
> a.b.c.101.  The idea would be that the VMs would appear on the real network. 
>   Then the 101 VM could migrate to a.b.c.2 and still be accessible.  I 
> envisioned some sort of proxy arp would happen so that every VM would simply 
> announce itself wherever it was.
> 
> This did seem to work in that I could ping from the VM:
> 
> # ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> 64 bytes from 8.8.8.8: icmp_seq=0 ttl=114 time=1.734 ms
> 
> Even IPV6:
> 
> # ping6 2605:2600:1001::4b
> PING6(56=40+8+8 bytes) 2605:2600:1001::4 --> 2605:2600:1001::4b
> 16 bytes from 2605:2600:1001::4b, icmp_seq=0 hlim=64 time=0.960 ms
> 16 bytes from 2605:2600:1001::4b, icmp_seq=1 hlim=64 time=0.415 ms
> 
> However TCP doesn't work.  In fact, I could only ping by IP because the 
> system couldn't connect to the DNS server, to get an address even though it 
> could ping it.
> 
> I guess my first question is does this seem doable?  If so, what am I 
> missing?  Is it possible that a bhyve switch is more like a router?

By switch, do you mean use bridge?  How specifically is the network
configured?

What you are describing sounds like what I do w/ bridge, but my use was
slightly more complicated.

Say your host has em0 as the main network, you would create a bridge0
interface, either via cloned_interfaces or via "ifconfig bridge0 create".
Then you would put the em0 interface as a member of the bridge
interface.  You would also add the tap interfaces of the various bhyve
vms as well (don't forget to make sure the tap interface is up on the
host, net.link.tap.up_on_open helps w/ this)...

I have heard (and that is the way I do that), that you have to put the
host IPs on the bridge0 interface, and not the em0 interface.

-- 
  John-Mark Gurney  Voice: +1 415 225 5579

 "All that I will do, has been done, All that I have, has not."


signature.asc
Description: PGP signature

When is a switch not a switch?

[D'Arcy Cain]

I am using bhyve with vm-bhyve,  I am trying to set up a virtual network 
with multiple hosts.  The idea is that a VM would be on the same virtual 
network no matter which actual host it is on.


Say I have a public network a.b.c.0/24.  I thought I could create a switch 
on a host.  The host would be a.b.c.1 and the VMs would be a.b.c.100 and 
a.b.c.101.  The idea would be that the VMs would appear on the real network. 
 Then the 101 VM could migrate to a.b.c.2 and still be accessible.  I 
envisioned some sort of proxy arp would happen so that every VM would simply 
announce itself wherever it was.


This did seem to work in that I could ping from the VM:

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=114 time=1.734 ms

Even IPV6:

# ping6 2605:2600:1001::4b
PING6(56=40+8+8 bytes) 2605:2600:1001::4 --> 2605:2600:1001::4b
16 bytes from 2605:2600:1001::4b, icmp_seq=0 hlim=64 time=0.960 ms
16 bytes from 2605:2600:1001::4b, icmp_seq=1 hlim=64 time=0.415 ms

However TCP doesn't work.  In fact, I could only ping by IP because the 
system couldn't connect to the DNS server, to get an address even though it 
could ping it.


I guess my first question is does this seem doable?  If so, what am I 
missing?  Is it possible that a bhyve switch is more like a router?


Thanks.

--
D'Arcy J.M. Cain  |  Democracy is three wolves
http://www.druid.net/darcy/|  and a sheep voting on
+1 416 788 2246 (DoD#0082)(eNTP)   |  what's for dinner.
IM: da...@vybenetworks.com, VoIP: sip:da...@druid.net

Disclaimer: By sending an email to ANY of my addresses you
are agreeing that:

1.  I am by definition, "the intended recipient".
2.  All information in the email is mine to do with as I see
fit and make such financial profit, political mileage, or
good joke as it lends itself to. In particular, I may quote
it where I please.
3.  I may take the contents as representing the views of
your company if I so wish.
4.  This overrides any disclaimer or statement of
confidentiality that may be included or implied in
your message.


OpenPGP_signature
Description: OpenPGP digital signature