[SOLVED] Re: Inter-VLAN routing on CURRENT: any known issues?

[O. Hartmann]

On Wed, 19 Jul 2017 16:44:16 +0300
Sergey Zhmylove  wrote:

To make things short:

Routing works as expected (even with the default route goinf via NAT).

The reason for the problems was: some in-hardware vlan feature support of the
i210/i350 chipset driver (or the chipset itself) seem to be broken.

I did not iterate deeply over the feature list, but I will soon, this is what
works so far for me at the moment with i210:

ifconfig_igbX="-vlanhwtso -vlanhwcsum -vlanhwfilter -vlanhwtag up"

Although I have disabled the feature "-vlanhwcsum" which the hardware
obviously provides, checking via ifconfig reveals:
igb1: flags=8943 metric 0 mtu
1500
options=6025ab

But this couldn't then be the culprit.

Some people mentioned earlier and suggested highly to disable those features -
I tried and put the minus-prepended disabling tag into rc.conf's
"create_args_igbX=..." - obviously not correct in that context.

Thank you very much for assisting!

Kind regards,

Oliver


> Do you receive packets from 192.168.2.0/24 and 192.168.3.0/24 on igb1.2 
> and igb1.3 respectively?
> Do you really need NAT? As far as I can see, you're looking for basic 
> static inter-VLAN routing.
> Could you check the communication between 2.0/24 and 3.0/24 with 
> unloaded ipfw module (just to exclude ipfw from the investigation)?
> 
> I have a lot of installations of such scheme on em(4) and re(4) devices 
> -- no problems at all. Even maybe there was igb(4) devices too.
> 
> Sergey  Zhmylove
> 17.07.2017 0:31, O. Hartmann пишет:
> > Am Sun, 16 Jul 2017 23:14:41 +0200
> > Frank Steinborn  schrieb:
> >  
> >> O. Hartmann  wrote:  
> >>> I have not have any success on this and I must ask now, to not make a
> >>> fool out of my self, whether the concept of having several vlan over one
> >>> single NIC is possible with FreeBSD (12-CURRENT, as of today, r321055.
> >>>
> >>> Since it is even not possible to "route" from a non-tagged igb1 to a
> >>> tagged vlan igb1.2 or igb1.66 (for instance) on the same NIC, I have a
> >>> faint suspect that I'm doing something terribly wrong.
> >>>
> >>> I think everyone working with vlan should have those problems, but since
> >>> I can not find anything on the list, I must do something wrong - my
> >>> simple conclusion.
> >>>
> >>> What is it?  
> >> Do you have enabled net.inet.ip.forwarding?
> >>  
> > Of course I have. As I stated earlier, ICMP pings from on VLAN to another
> > over this router works, but any IP (UDP, TCP) is vanishing into thin air.
> >
> > I don't have a FBSD-11-STABLE reference system at hand, so that I can check
> > with another revision/major release of the OS, but I work on that.
> >  
> 
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Inter-VLAN routing on CURRENT: any known issues?

[Sergey Zhmylove]

Do you receive packets from 192.168.2.0/24 and 192.168.3.0/24 on igb1.2 
and igb1.3 respectively?
Do you really need NAT? As far as I can see, you're looking for basic 
static inter-VLAN routing.
Could you check the communication between 2.0/24 and 3.0/24 with 
unloaded ipfw module (just to exclude ipfw from the investigation)?


I have a lot of installations of such scheme on em(4) and re(4) devices 
-- no problems at all. Even maybe there was igb(4) devices too.


Sergey  Zhmylove
17.07.2017 0:31, O. Hartmann пишет:

Am Sun, 16 Jul 2017 23:14:41 +0200
Frank Steinborn  schrieb:


O. Hartmann  wrote:

I have not have any success on this and I must ask now, to not make a fool out 
of my
self, whether the concept of having several vlan over one single NIC is 
possible with
FreeBSD (12-CURRENT, as of today, r321055.

Since it is even not possible to "route" from a non-tagged igb1 to a tagged vlan
igb1.2 or igb1.66 (for instance) on the same NIC, I have a faint suspect that 
I'm
doing something terribly wrong.

I think everyone working with vlan should have those problems, but since I can 
not
find anything on the list, I must do something wrong - my simple conclusion.

What is it?

Do you have enabled net.inet.ip.forwarding?


Of course I have. As I stated earlier, ICMP pings from on VLAN to another over 
this
router works, but any IP (UDP, TCP) is vanishing into thin air.

I don't have a FBSD-11-STABLE reference system at hand, so that I can check 
with another
revision/major release of the OS, but I work on that.



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Inter-VLAN routing on CURRENT: any known issues?

[O. Hartmann]

Am Sun, 16 Jul 2017 23:14:41 +0200
Frank Steinborn  schrieb:

> O. Hartmann  wrote:
> > I have not have any success on this and I must ask now, to not make a fool 
> > out of my
> > self, whether the concept of having several vlan over one single NIC is 
> > possible with
> > FreeBSD (12-CURRENT, as of today, r321055.
> > 
> > Since it is even not possible to "route" from a non-tagged igb1 to a tagged 
> > vlan
> > igb1.2 or igb1.66 (for instance) on the same NIC, I have a faint suspect 
> > that I'm
> > doing something terribly wrong.
> > 
> > I think everyone working with vlan should have those problems, but since I 
> > can not
> > find anything on the list, I must do something wrong - my simple conclusion.
> > 
> > What is it?  
> 
> Do you have enabled net.inet.ip.forwarding?
> 

Of course I have. As I stated earlier, ICMP pings from on VLAN to another over 
this
router works, but any IP (UDP, TCP) is vanishing into thin air.

I don't have a FBSD-11-STABLE reference system at hand, so that I can check 
with another
revision/major release of the OS, but I work on that. 

-- 
O. Hartmann

Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).


pgpsFj63_E5Rs.pgp
Description: OpenPGP digital signature

Re: Inter-VLAN routing on CURRENT: any known issues?

[O. Hartmann]

Am Sun, 16 Jul 2017 23:14:41 +0200
Frank Steinborn  schrieb:

> O. Hartmann  wrote:
> > I have not have any success on this and I must ask now, to not make a fool 
> > out of my
> > self, whether the concept of having several vlan over one single NIC is 
> > possible with
> > FreeBSD (12-CURRENT, as of today, r321055.
> > 
> > Since it is even not possible to "route" from a non-tagged igb1 to a tagged 
> > vlan
> > igb1.2 or igb1.66 (for instance) on the same NIC, I have a faint suspect 
> > that I'm
> > doing something terribly wrong.
> > 
> > I think everyone working with vlan should have those problems, but since I 
> > can not
> > find anything on the list, I must do something wrong - my simple conclusion.
> > 
> > What is it?  
> 
> Do you have enabled net.inet.ip.forwarding?
> 


... aber selbstverständlich doch, das ist das erste seit nunmehr 20 Jahren 
FreeBSD, was
auf einem Router eingestellt wird ...

Wie ich bereits beschrieben habe: pingen (ICMP) auf andere VLANs geht, nur 
keine IP
Dienste - und das mit oder ohne eingeschalteter IPFW.

Grüße,

oh



-- 
O. Hartmann

Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).


pgp5SoJDaNM66.pgp
Description: OpenPGP digital signature

Re: Inter-VLAN routing on CURRENT: any known issues?

[Frank Steinborn]

O. Hartmann  wrote:
> I have not have any success on this and I must ask now, to not make a fool 
> out of my
> self, whether the concept of having several vlan over one single NIC is 
> possible with
> FreeBSD (12-CURRENT, as of today, r321055.
> 
> Since it is even not possible to "route" from a non-tagged igb1 to a tagged 
> vlan igb1.2
> or igb1.66 (for instance) on the same NIC, I have a faint suspect that I'm 
> doing
> something terribly wrong.
> 
> I think everyone working with vlan should have those problems, but since I 
> can not find
> anything on the list, I must do something wrong - my simple conclusion.
> 
> What is it?

Do you have enabled net.inet.ip.forwarding?

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Inter-VLAN routing on CURRENT: any known issues?

[O. Hartmann]

Am Fri, 14 Jul 2017 15:00:30 +0300
"Andrey V. Elsukov"  schrieb:

> On 14.07.2017 14:42, O. Hartmann wrote:
> > I use in-kernel NAT. IPFW is performing NAT. In firewall type "OPEN" from 
> > the
> > vanilla rc.conf, IPFW has instance "nat 123" which provides then NAT.  
> 
> I never used default config types for firewall, so, it would be nice to
> see what rules do you have.
> 
> # ipfw show
> # ipfw nat show config
> 
> >> VLANs work on the layer2  
> > According to 1):
> > 
> > I consider the settings of the switch now as correct. I have no access to 
> > the
> > router right now. But I did short experiments yesterday evening and it is
> > weird: loged in on thr router, I can ping every host on any VLAN, so ICMP
> > travel from the router the right way to its destination and back.
> > 
> > From any host on any VLAN that is "trunked" through the router, I can ping 
> > any
> > other host on any other VLAN, preferrably not on the same VLAN. By cutting 
> > off
> > the trunk line to the router, pinging stops immediately.
> > 
> > From any host on any VLAN I can ping any host which is NATed on the outside
> > world.
> > 
> > From the router itself, I can ssh into any host on any VLAN providing ssh
> > service. That said, according to question 3), NAT is considered to be setup
> > correctly.
> > 
> > Now the strange things: Neither UDP, nor TCP services "flow" from hosts on 
> > one
> > VLAN to hosts on a different VLAN. Even ssh doens't work. 
> > When loged in onto the router, I can't "traceroute" any host on any VLAN.  
> 
> This is most likely due to the problem with firewall rules.
> If you set net.inet.ip.firewall.enable=0, does it solve the problem with
> TCP/UDP between hosts on a different VLANs?
> 
> > According to question 2), the ability to ping from, say, a host on VLAN 
> > 1000 to
> > another host on VLAN 2 passing through the router would indicate that both
> > sides know their routes to each other. Or am I wrong?  
> 
> Yes.
> 
> > I got words from Sean bruno that there might be a problem with the Intel 
> > i210
> > chipset in recent CURRENT - and the hardware on the PCEngine APU 2C4 is 
> > three
> > i210. I'm aware of the problem since r320134 (the oldest CURRENT I started
> > experimenting with the VLAN trunking).  
> 
> It is very strange problems, why ICMP works, but TCP/UDP does not? :)
> You can try to disable any type of offloading for the card, there were
> some problems in the past with checksum offlading, that may lead to the
> problems with TCP, but this usually should be noticeable in the tcpdump
> output.
> 

I have not have any success on this and I must ask now, to not make a fool out 
of my
self, whether the concept of having several vlan over one single NIC is 
possible with
FreeBSD (12-CURRENT, as of today, r321055.

Since it is even not possible to "route" from a non-tagged igb1 to a tagged 
vlan igb1.2
or igb1.66 (for instance) on the same NIC, I have a faint suspect that I'm doing
something terribly wrong.

I think everyone working with vlan should have those problems, but since I can 
not find
anything on the list, I must do something wrong - my simple conclusion.

What is it?

-- 
O. Hartmann

Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).


pgpDBY3Opn8QH.pgp
Description: OpenPGP digital signature

Re: Inter-VLAN routing on CURRENT: any known issues?

[O. Hartmann]

Am Fri, 14 Jul 2017 15:00:30 +0300
"Andrey V. Elsukov"  schrieb:

> On 14.07.2017 14:42, O. Hartmann wrote:
> > I use in-kernel NAT. IPFW is performing NAT. In firewall type "OPEN" from 
> > the
> > vanilla rc.conf, IPFW has instance "nat 123" which provides then NAT.  
> 
> I never used default config types for firewall, so, it would be nice to
> see what rules do you have.

Me neither except on some hosts with very little complications in their setups 
or simple
clients.

> 
> # ipfw show

The OPEN firewall rules, which show the very same behaviour as I stated before:

root@gate:~ # ipfw list
00050 nat 123 ip4 from any to any via tun0
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
65000 allow ip from any to any
65535 deny ip from any to any

> # ipfw nat show config

root@gate:~ # ipfw nat show config
ipfw nat 123 config if tun0 log

or

ipfw nat 1 config if tun0 log same_ports reset redirect_port tcp 
192.168.0.111:9734 9734
redirect_port tcp 192.168.0.111:5432 5432 redirect_port udp 192.168.2.1:2427 
2427
redirect_port udp 192.168.2.1:4569 4569 redirect_port udp 192.168.2.1:5060 5060
redirect_port tcp 192.168.2.1:5060 5060 redirect_port tcp 192.168.0.111:443 443
redirect_port tcp 192.168.0.111:80 80 redirect_port tcp 192.168.0.111:22 22

> 
> >> VLANs work on the layer2  
> > According to 1):
> > 
> > I consider the settings of the switch now as correct. I have no access to 
> > the
> > router right now. But I did short experiments yesterday evening and it is
> > weird: loged in on thr router, I can ping every host on any VLAN, so ICMP
> > travel from the router the right way to its destination and back.
> > 
> > From any host on any VLAN that is "trunked" through the router, I can ping 
> > any
> > other host on any other VLAN, preferrably not on the same VLAN. By cutting 
> > off
> > the trunk line to the router, pinging stops immediately.
> > 
> > From any host on any VLAN I can ping any host which is NATed on the outside
> > world.
> > 
> > From the router itself, I can ssh into any host on any VLAN providing ssh
> > service. That said, according to question 3), NAT is considered to be setup
> > correctly.
> > 
> > Now the strange things: Neither UDP, nor TCP services "flow" from hosts on 
> > one
> > VLAN to hosts on a different VLAN. Even ssh doens't work. 
> > When loged in onto the router, I can't "traceroute" any host on any VLAN.  
> 
> This is most likely due to the problem with firewall rules.
> If you set net.inet.ip.firewall.enable=0, does it solve the problem with
> TCP/UDP between hosts on a different VLANs?

net.inet.ip.firewall.enable does not exist, I suppose it is 
net.inet.ip.fw.enable.

Not, it doesn't change anything, last rule in my list is deny all, as you can 
see above
(in-kernel).

> 
> > According to question 2), the ability to ping from, say, a host on VLAN 
> > 1000 to
> > another host on VLAN 2 passing through the router would indicate that both
> > sides know their routes to each other. Or am I wrong?  
> 
> Yes.
> 
> > I got words from Sean bruno that there might be a problem with the Intel 
> > i210
> > chipset in recent CURRENT - and the hardware on the PCEngine APU 2C4 is 
> > three
> > i210. I'm aware of the problem since r320134 (the oldest CURRENT I started
> > experimenting with the VLAN trunking).  
> 
> It is very strange problems, why ICMP works, but TCP/UDP does not? :)
> You can try to disable any type of offloading for the card, there were
> some problems in the past with checksum offlading, that may lead to the
> problems with TCP, but this usually should be noticeable in the tcpdump
> output.
> 

I tried that, but somehow I do not have any check:

ifconfig_igb1="up"
#ifconfig_igb1="inet6 ::1 prefixlen 64 mtu 6121"
create_args_igb1="-tso -lro -rxcsum -txcsum -rxcsum6 -txcsum6 -vlanhwtso 
-vlanhwcsum
-vlanhwfilter -vlanhwtag"


and ifconfig igb1:

igb1: flags=8843 metric 0 mtu 1500

options=6525bb



Kind regards,

Oliver
-- 
O. Hartmann

Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).


pgpMLW2LojGUl.pgp
Description: OpenPGP digital signature

Re: Inter-VLAN routing on CURRENT: any known issues?

[Andrey V. Elsukov]

On 14.07.2017 14:42, O. Hartmann wrote:
> I use in-kernel NAT. IPFW is performing NAT. In firewall type "OPEN" from the
> vanilla rc.conf, IPFW has instance "nat 123" which provides then NAT.

I never used default config types for firewall, so, it would be nice to
see what rules do you have.

# ipfw show
# ipfw nat show config

>> VLANs work on the layer2
> According to 1):
> 
> I consider the settings of the switch now as correct. I have no access to the
> router right now. But I did short experiments yesterday evening and it is
> weird: loged in on thr router, I can ping every host on any VLAN, so ICMP
> travel from the router the right way to its destination and back.
> 
> From any host on any VLAN that is "trunked" through the router, I can ping any
> other host on any other VLAN, preferrably not on the same VLAN. By cutting off
> the trunk line to the router, pinging stops immediately.
> 
> From any host on any VLAN I can ping any host which is NATed on the outside
> world.
> 
> From the router itself, I can ssh into any host on any VLAN providing ssh
> service. That said, according to question 3), NAT is considered to be setup
> correctly.
> 
> Now the strange things: Neither UDP, nor TCP services "flow" from hosts on one
> VLAN to hosts on a different VLAN. Even ssh doens't work. 
> When loged in onto the router, I can't "traceroute" any host on any VLAN.

This is most likely due to the problem with firewall rules.
If you set net.inet.ip.firewall.enable=0, does it solve the problem with
TCP/UDP between hosts on a different VLANs?

> According to question 2), the ability to ping from, say, a host on VLAN 1000 
> to
> another host on VLAN 2 passing through the router would indicate that both
> sides know their routes to each other. Or am I wrong?

Yes.

> I got words from Sean bruno that there might be a problem with the Intel i210
> chipset in recent CURRENT - and the hardware on the PCEngine APU 2C4 is three
> i210. I'm aware of the problem since r320134 (the oldest CURRENT I started
> experimenting with the VLAN trunking).

It is very strange problems, why ICMP works, but TCP/UDP does not? :)
You can try to disable any type of offloading for the card, there were
some problems in the past with checksum offlading, that may lead to the
problems with TCP, but this usually should be noticeable in the tcpdump
output.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature

Re: Inter-VLAN routing on CURRENT: any known issues?

[O. Hartmann]

Am Thu, 13 Jul 2017 16:12:06 +0300
"Andrey V. Elsukov"  schrieb:

> On 12.07.2017 22:43, O. Hartmann wrote:
> > Now the FUN PART:
> > 
> > From any host in any VLAN I'm able to ping hosts on the wild internet via
> > their IP, on VLAN 1000 there is a DNS running, so I'm also able to resolv
> > names like google.com or FreeBSD.org. But I can NOT(!) access any host via
> > http/www or ssh.   
> 
> You have not specified where is the NAT configured and its settings is
> matters.

I use in-kernel NAT. IPFW is performing NAT. In firewall type "OPEN" from the
vanilla rc.conf, IPFW has instance "nat 123" which provides then NAT.

> 
> VLANs work on the layer2, they do not used for IP routing. Each received
> packet loses its layer2 header before it gets taken by IP stack. If an
> IP packet should be routed, the IP stack determines outgoing interface
> and new ethernet header with VLAN header from this interface is prepended.

Since all VLANs are on the same NIC on that router, they should only differ in
the VLAN tag.

> 
> What I would do in your place:
> 1. Check the correctness of the switch settings.
>   - on the router use tcpdump on each vlan interface and
> also directly on igb1. Use -e argument to see ethernet header.
> Try ping router's IP address from each vlan, you should see tagged
> packet on igb1 and untagged on corresponding vlan interface.
> 
> 2. Check the correctness of the routing settings for each used node.
>   - to be able establish connection from one vlan to another, both nodes
> must have a route to each other.
> 
> 3. Check the NAT settings.
>   - to be able to connect to the Internet from your addresses, you must
> use NAT. If you don't have NAT, but it somehow works, this means
> that some device does the translation for you, but it's
> configuration does not meet to your requirements. And probably you
> need to translate prefixes configured for your vlans independently.
> 

According to 1):

I consider the settings of the switch now as correct. I have no access to the
router right now. But I did short experiments yesterday evening and it is
weird: loged in on thr router, I can ping every host on any VLAN, so ICMP
travel from the router the right way to its destination and back.

From any host on any VLAN that is "trunked" through the router, I can ping any
other host on any other VLAN, preferrably not on the same VLAN. By cutting off
the trunk line to the router, pinging stops immediately.

From any host on any VLAN I can ping any host which is NATed on the outside
world.

From the router itself, I can ssh into any host on any VLAN providing ssh
service. That said, according to question 3), NAT is considered to be setup
correctly.

Now the strange things: Neither UDP, nor TCP services "flow" from hosts on one
VLAN to hosts on a different VLAN. Even ssh doens't work. 
When loged in onto the router, I can't "traceroute" any host on any VLAN.

According to question 2), the ability to ping from, say, a host on VLAN 1000 to
another host on VLAN 2 passing through the router would indicate that both
sides know their routes to each other. Or am I wrong?

I got words from Sean bruno that there might be a problem with the Intel i210
chipset in recent CURRENT - and the hardware on the PCEngine APU 2C4 is three
i210. I'm aware of the problem since r320134 (the oldest CURRENT I started
experimenting with the VLAN trunking).

I hope it might be a problem with the driver, otherwise I have fully
misunderstood FreeBSD's network abilities and techniques :-(

I'll provide tcpdump data later.

Kind regards,

Oliver 


-- 
O. Hartmann

Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Inter-VLAN routing on CURRENT: any known issues?

[Andrey V. Elsukov]

On 12.07.2017 22:43, O. Hartmann wrote:
> Now the FUN PART:
> 
> From any host in any VLAN I'm able to ping hosts on the wild internet via 
> their IP, on
> VLAN 1000 there is a DNS running, so I'm also able to resolv names like 
> google.com or
> FreeBSD.org. But I can NOT(!) access any host via http/www or ssh. 

You have not specified where is the NAT configured and its settings is
matters.

VLANs work on the layer2, they do not used for IP routing. Each received
packet loses its layer2 header before it gets taken by IP stack. If an
IP packet should be routed, the IP stack determines outgoing interface
and new ethernet header with VLAN header from this interface is prepended.

What I would do in your place:
1. Check the correctness of the switch settings.
  - on the router use tcpdump on each vlan interface and
also directly on igb1. Use -e argument to see ethernet header.
Try ping router's IP address from each vlan, you should see tagged
packet on igb1 and untagged on corresponding vlan interface.

2. Check the correctness of the routing settings for each used node.
  - to be able establish connection from one vlan to another, both nodes
must have a route to each other.

3. Check the NAT settings.
  - to be able to connect to the Internet from your addresses, you must
use NAT. If you don't have NAT, but it somehow works, this means
that some device does the translation for you, but it's
configuration does not meet to your requirements. And probably you
need to translate prefixes configured for your vlans independently.

-- 
WBR, Andrey V. Elsukov



signature.asc
Description: OpenPGP digital signature