Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

> I gave up and looked at some code.  :-)  The x200C is generated when you try 
> to register a universal MAC address or one that potentially conflicts with a 
> MAC address CP might create.
> That means you can only register additional MAC addresses that have MAC 
> prefixes that are outside the range of the SYSTEM or USER MACPREFIX 
> identified in SYSTEM CONFIG.  

Thanks, this explains a lot. Knowing this I was actually able to register a 
secondary mac address with the vswitch in my experimental setup.

> This opens a dangerous door because it means that the non-bridge guests must 
> also have MACPROTECT OFF on their NICs.
> Further, you can't manage the MAC addresses in CP; the MAC address has to 
> appear in the Linux configuration (hwaddr).

This solves the problem for the likes of KVM where the hypervisor inside Linux 
can manage its own set of mac addresses. It does not solve the problem for us, 
because we rely on CP managing the mac addresses (i.e. we do not want the guest 
OSes to have the ability to change their mac address). But this is very useful 
information. We will most likely go into redesign phase, but now we know much 
more about what are the limits that we operate under.

> I think I would raise a PMR to discuss this with z/VM Development.
We actually have an open PMR but at this point in time I have learned more from 
these discussions, so thanks again.

Tomas

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Alan Altmark]

On Wednesday, 02/26/2014 at 08:44 EST, "Pavelka, Tomas"
 wrote:
> The dbf trace shows that the l2 qeth driver code gets exercised and
leads to a
> command called IPA_CMD_SETVMAC. The user mode call succeeds but I have
found
> this error message in /sys/kernel/debug/s390dbf/qeth_msg/sprintf
>
> 00 01393420032:989799 2 - 00 03c00092471e  IPA: setvmac(x21) for
> 0.0.a100/eth7 returned x200C "L2 mac not authorized by hypervisor"
>
> This is where we are stuck again. Is the secondary unicast address what
I think
> it is? Are there any restrictions on in what situations secondary MAC
addresses
> can be added to a virtual NIC?

I gave up and looked at some code.  :-)  The x200C is generated when you
try to register a universal MAC address or one that potentially conflicts
with a MAC address CP might create.

That means you can only register additional MAC addresses that have MAC
prefixes that are outside the range of the SYSTEM or USER MACPREFIX
identified in SYSTEM CONFIG.  This opens a dangerous door because it means
that the non-bridge guests must also have MACPROTECT OFF on their NICs.
Further, you can't manage the MAC addresses in CP; the MAC address has to
appear in the Linux configuration (hwaddr).

The core design points you are bumping into are:
1.  CP's requirement that MAC addresses be unique within an LPAR, while
network devices only require that they be unique within a LAN segment or
VLAN.
2.  The inability to specify "outlander" MAC addresses in their 6-byte
entirety on NICDEF & Co., which would enable MACPROTECT ON to be engaged.

I think I would raise a PMR to discuss this with z/VM Development.

Alan Altmark

Senior Managing z/VM and Linux Consultant
IBM System Lab Services and Training
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

We have done more research and found out that the same problem as we have is 
faced by virtualization software running under Linux. I have found a discussion 
about a feature called "secondary unicast address" that would allow registering 
multiple MAC addresses with a NIC. Here is a discussion that explains why this 
is useful for virtualization:
http://thread.gmane.org/gmane.linux.network/64719/focus=64775
I have been looking at the l2 qeth driver and found out that the code seems to 
support secondary unicast MAC addresses. Adding a secondary unicast address to 
a NIC can actually be done from user mode, here is a code snippet that 
registers a secondary MAC via a raw packet interface:

s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL));

...

struct packet_mreq mreq;
memset(&mreq, 0, sizeof(mreq));
mreq.mr_ifindex=ifr.ifr_ifindex;
mreq.mr_type=PACKET_MR_UNICAST;
mreq.mr_alen=ETHER_ADDR_LEN;
memcpy(mreq.mr_address,mac,ETHER_ADDR_LEN);
ret = setsockopt(s, SOL_PACKET, PACKET_ADD_MEMBERSHIP, &mreq, sizeof(mreq));

The dbf trace shows that the l2 qeth driver code gets exercised and leads to a 
command called IPA_CMD_SETVMAC. The user mode call succeeds but I have found 
this error message in /sys/kernel/debug/s390dbf/qeth_msg/sprintf

00 01393420032:989799 2 - 00 03c00092471e  IPA: setvmac(x21) for 
0.0.a100/eth7 returned x200C "L2 mac not authorized by hypervisor"

This is where we are stuck again. Is the secondary unicast address what I think 
it is? Are there any restrictions on in what situations secondary MAC addresses 
can be added to a virtual NIC?

Tomas

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

I was talking to Carsten and he said that the MAC address gets registered with 
the OSA card via calls from the Linux kernel drivers. He also mentioned that 
KVM faces the same problem. We will attempt to do the registering of the extra 
MAC addresses from a kernel module running on the bridge machine. I'll let you 
know if we have any success with that.

Tomas

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Alan Altmark]

On Friday, 02/21/2014 at 03:38 EST, Mark Post  wrote:
> >>> On 2/21/2014 at 02:42 PM, Alan Altmark 
wrote:
> > However, that only works for virtual machines -- you cannot act as a
> > bridge for external traffic.  The reason is that CP allows only one
> > virtual NIC to be assigned a particular MAC address at one time.   If
the
> > bridge sends a frame to an unregistered MAC, CP will send that frame
out
> > on the wire, but if the external host responds, the OSA will not
forward
> > the inbound frame *because* the MAC isn't registered.
>
> Yes, this is what I was seeing when testing KVM on System z.  The OSA is
the
> cause of the traffic not getting passed on to the originating guest
(whether
> z/VM or KVM).  I suppose if you used something like OSA/SF (which I have
zero
> experience with, so don't get your hopes up) to register the MAC with
the OSA
> that things might work.  Kind of unreasonable to expect customers to do
that
> for every guest that gets created on a system.

To see what you can do, you have to go into OSA Advanced Facilities on the
SE.  OSA-Express 4s was the last OSA to support configuration via OSA/SF.
(Like the 5s, it can also be managed by the SE).

Whether dedicated OSA or vNIC, a guest can register multiple MACs (with
MACPROTECT OFF) just as the VSWITCH does.  Whether kvm will do that or
not, I don't know.

Alan Altmark

Senior Managing z/VM and Linux Consultant
IBM System Lab Services and Training
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Mark Post]

>>> On 2/21/2014 at 02:42 PM, Alan Altmark  wrote: 
> However, that only works for virtual machines -- you cannot act as a
> bridge for external traffic.  The reason is that CP allows only one
> virtual NIC to be assigned a particular MAC address at one time.   If the
> bridge sends a frame to an unregistered MAC, CP will send that frame out
> on the wire, but if the external host responds, the OSA will not forward
> the inbound frame *because* the MAC isn't registered.

Yes, this is what I was seeing when testing KVM on System z.  The OSA is the 
cause of the traffic not getting passed on to the originating guest (whether 
z/VM or KVM).  I suppose if you used something like OSA/SF (which I have zero 
experience with, so don't get your hopes up) to register the MAC with the OSA 
that things might work.  Kind of unreasonable to expect customers to do that 
for every guest that gets created on a system.


Mark Post

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Alan Altmark]

On Friday, 02/21/2014 at 04:54 EST, "Pavelka, Tomas"
 wrote:
> I hope we are not developing on top of a bug exploit. Let me know if I
should
> add more details about the setup.

Tomas, I did some more research and you are fine.  Sort of.  Unless you
have MACPROTECT ON (VMLAN statement, VSWITCH definition, NICDEF, or SET
NIC), a guest using a layer 2 VSWITCH can emit frames using whatever
origin MAC address they wish, including spoofing other guest MACs.  With
MACPROTECT ON (I recommend that everyone set that as the default in SYSTEM
CONFIG), the guest can only use the origin MAC address that CP has
assigned.

However, that only works for virtual machines -- you cannot act as a
bridge for external traffic.  The reason is that CP allows only one
virtual NIC to be assigned a particular MAC address at one time.   If the
bridge sends a frame to an unregistered MAC, CP will send that frame out
on the wire, but if the external host responds, the OSA will not forward
the inbound frame *because* the MAC isn't registered.

Regards,
Alan Altmark

Senior Managing z/VM and Linux Consultant
IBM System Lab Services and Training
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

> so you must be performing MAC address translation such that you look more 
> like a a layer 2 router (a la OSA in layer 3 mode), 
> not an 802.1d bridge (I said 802.3 earlier; I meant 802.1d.)  That is, all 
> guests on the PUBLIC vswitch have the same MAC 
> address as viewed by all hosts on the PRIVATE vswitch (and vice versa).

In my original question I changed the names of vswitches and virtual machines. 
This one will have the real names which are not as self-explanatory because I'm 
using what is available. Here are more details on the setup:

AL070009 (141.202.59.44/02-00-C2-00-01-3A) -  - TOMRH61 (runs bridge) - 
 - AL07000A (141.202.59.45/02-00-C2-00-01-3B)

I ran a ping from AL070009

PING 141.202.59.45 (141.202.59.45) 56(84) bytes of data.
64 bytes from 141.202.59.45: icmp_seq=1 ttl=64 time=1.40 ms
64 bytes from 141.202.59.45: icmp_seq=2 ttl=64 time=0.426 ms

And three TCP dumps:

First one at the interface on AL070009:

03:21:01.089237 02:00:c2:00:01:3a (oui Unknown) > 02:00:c2:00:01:3b (oui 
Unknown), ethertype IPv4 (0x0800), length 98: 141.202.59.44 > 141.202.59.45: 
ICMP echo request, id 2040, seq 1, length 64

Second one on the bridge (TOMRH61):

03:21:01.089304 02:00:c2:00:01:3a (oui Unknown) > 02:00:c2:00:01:3b (oui 
Unknown), ethertype IPv4 (0x0800), length 98: 141.202.59.44 > 141.202.59.45: 
ICMP echo request, id 2040, seq 1, length 64

And third one at the interface at the destination guest (AL07000A):

03:21:01.090045 02:00:c2:00:01:3a (oui Unknown) > 02:00:c2:00:01:3b (oui 
Unknown), ethertype IPv4 (0x0800), length 98: 141.202.59.44 > 141.202.59.45: 
ICMP echo request, id 2040, seq 1, length 64

As sanity check I have saved the results of CP Q VSWITCH ACT for both vswitches 
and verified that ALBL07 (private vswitch) does not have 02-00-C2-00-01-3B 
registered and that INTRA59 (public vswitch) does not have 02-00-C2-00-01-3A 
registered:

+ grep 02-00-C2-00-01-3A dumps/private_switch.act
  02-00-C2-00-01-3A IP: 141.202.59.44
+ grep 02-00-C2-00-01-3B dumps/private_switch.act
+ grep 02-00-C2-00-01-3A dumps/public_switch.act
+ grep 02-00-C2-00-01-3B dumps/public_switch.act
  02-00-C2-00-01-3B IP: 141.202.59.45

As a second sanity check I have revoked access of AL07000A from all vswitches 
except for INTRA59 which left the Linux machine with a single active NIC with 
MAC 02-00-C2-00-01-3B (I used another vswitch for SSH access). I was still able 
to ping in that scenario.

>From this I think that sending MAC addresses unknown to a vswitch is possible 
>and the vswitch will not block it. A frame with src mac address 
>02-00-C2-00-01-3A which is only known on ALBL07 gets received by a guest only 
>connected to INTRA59 which does not know the source mac address. I do not 
>think we have duplicate MAC addresses on the whole z/VM system.

The bridge software I use is this: 
http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge which 
they say implements a subset of ANSI/IEEE 802.1d.

I hope we are not developing on top of a bug exploit. Let me know if I should 
add more details about the setup.

Thanks,
Tomas

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

>so you must be performing MAC address translation such that you look more like 
>a a layer 2 router (a la OSA in layer 3 mode), not an 802.1d bridge (I said 
>802.3 earlier; I meant
802.1d.)  That is, all guests on the PUBLIC vswitch have the same MAC address 
as viewed by all hosts on the PRIVATE vswitch (and vice versa).

I don't think we do any translation. But I want to ensure that what I think is 
in fact consistent with reality ;-) I will rerun the experiment and post 
details of vswitch settings and TCP dumps to show what the MAC addresses are at 
various places in the communication.

Thanks,
Tomas

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Alan Altmark]

On Thursday, 02/20/2014 at 09:33 EST, "Pavelka, Tomas"
 wrote:
> > Further, the VSWITCH is already acting as an IEEE 802.3 layer 2 bridge
and
> its filtering database will drop unicast frames destined for unknown MAC
> addresses.
>
> One thing I forgot to mention: We have successfully sent packets between
two
> vswitches connected to a Linux bridge (LINUX1 and LINUX2 communicate in
the
> example below). But we needed to put the Linux bridge into promiscuous
mode on
> both of the bridged vswitches.
>
> (LINUX1) -  - (LINUXBR) -  - (LINUX2)

I thought about that, but being in promiscuous mode doesn't change the
rules for LINUXBR's origin MAC address, so you must be performing MAC
address translation such that you look more like a a layer 2 router (a la
OSA in layer 3 mode), not an 802.1d bridge (I said 802.3 earlier; I meant
802.1d.)  That is, all guests on the PUBLIC vswitch have the same MAC
address as viewed by all hosts on the PRIVATE vswitch (and vice versa).

So just beware that anything that assumes a unique MAC address per host
(typically DHCP) will not operate correctly across your bridge.

Alan Altmark

Senior Managing z/VM and Linux Consultant
IBM System Lab Services and Training
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

> Further, the VSWITCH is already acting as an IEEE 802.3 layer 2 bridge and 
> its filtering database will drop unicast frames destined for unknown MAC 
> addresses.

One thing I forgot to mention: We have successfully sent packets between two 
vswitches connected to a Linux bridge (LINUX1 and LINUX2 communicate in the 
example below). But we needed to put the Linux bridge into promiscuous mode on 
both of the bridged vswitches.

(LINUX1) -  - (LINUXBR) -  - (LINUX2)

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

> What is LINUXBR doing for you that the VSWITCH cannot do for you?

We are in the business of porting software that works on top of a Linux bridge. 
We have a kernel driver that hooks into the Linux bridge and filters layer 2 
frames based on rules. We got it working on Xen, VMware and inside z/VM (Linux 
bridging between vswitches). The OSA problem caught us by surprise (namely the 
fact that OSA behaves differently than a vswitch otherwise we would have 
discovered this sooner).

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Alan Altmark]

On Thursday, 02/20/2014 at 03:08 EST, "Pavelka, Tomas"
 wrote:
> We have a problem where frames that pass through a Linux bridge do not
reach
> the gateway outside of the mainframe box. We have set up an experiment
that
> reproduces the problem, which looks like this:
>
> (LINUX1) -  - (LINUXBR) -  - OSA -
gateway
>
> The problem is that in this setup we cannot ping the gateway. But, under
a
> different setup:
>
> (LINUX1) -  - (LINUXBR) -  - (LINUX2)
>
> Both LINUX1 and LINUX2 can communicate. Moreover, LINUX2 can ping the
gateway
> (the OSA card is still connected to the public vswitch, I just did not
put it
> in the picture).
>
> Some more details that may be important:
> - Both public and private vswitch are layer 2
> - LINUXBR runs RHEL 6 and uses bridge-utils to create the bridge
> - private vswitch is not connected to any OSA card
>
> We have played with TCPDUMP and found that ARP (broadcast) packets do
reach the
> gateway and come back, but ping's ICMP (unicast) packets get dropped.
This led
> us to the following hypothesis: If there is a unicast packet originating
from a
> MAC address not known to public vswitch, it gets dropped somewhere on
the way
> between LINUXBR and the gateway.
>
> Does anyone know any settings that could affect filtering done either by
the
> vswitch or by the OSA card? We asked our hardware people but they did
not know
> of anything that could cause the problems. But a more targeted question
could
> help if we knew what to ask for.

Two guests cannot use the same MAC address, even if they are on different
VSWITCHes.   This is true even if you permit the bridging guest to set its
own MAC address (MACPROTECT OFF).   [NOTE:  Everyone should have VMLAN
MACPROTECT ON in SYSTEM CONFIG.]  So you cannot implement a layer 2 bridge
with VSWITCHes.

Further, the VSWITCH is already acting as an IEEE 802.3 layer 2 bridge and
its filtering database will drop unicast frames destined for unknown MAC
addresses.

What is LINUXBR doing for you that the VSWITCH cannot do for you?

Alan Altmark

Senior Managing z/VM and Linux Consultant
IBM System Lab Services and Training
ibm.com/systems/services/labservices
office: 607.429.3323
mobile; 607.321.7556
alan_altm...@us.ibm.com
IBM Endicott

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Berthold Gunreben]

I did not really try, but what you probably could do, is using MAC
Masquerading with ebtables on your LINUXBR machine.
http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html

Berthold

On Thu, 20 Feb 2014 09:35:00 +0100
Carsten Otte  wrote:

> This setup won't work, because Linux negotiates its mac address with
> the OSA, and cannot send frames from another mac. You could use ip
> forwarding, and have Linux route on layer 3. This should work, as
> long as you use the OSA
> in layer 2 mode.
> 

-- 
--
 Berthold Gunreben  Build Service Team
 http://www.suse.de/ Maxfeldstr. 5
 SUSE LINUX Products GmbH   D-90409 Nuernberg, Germany
 GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer
 HRB 16746 (AG Nürnberg)

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Carlos Romero Martin]

On Thu, 20 Feb 2014 09:44:45 +, Pavelka, Tomas wrote:

Another question that comes to mind is, if there is negotiation with
OSA, how does Linux tell that there is a real OSA involved? My
assumptions (which may be false ;-)) were that Linux as a z/VM guest
should not be able to tell whether a NIC is real or virtual. And in
our case the NIC is always virtual, because we do not connect
directly
to the OSA, we go through a vswitch. The bridging works as long as
the
traffic does not go through the OSA card, so somehow the Linux guest
is able to pass frames with MAC addresses it does not own. These
frames are only dropped if they go towards the OSA. I was not able to
tell whether they are dropped by the OSA or by the vswitch which
connects to the OSA. But the same vswitch passes the bridged packets
with no problems to other Linux machines inside the box...


--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390
or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390

--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Hello all,

You have a RedBooks - OSA-Express Implementation Guide
http://www.redbooks.ibm.com/redpieces/abstracts/sg245948.html?Open

Best regards

Carlos Romero Martin

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

Another question that comes to mind is, if there is negotiation with OSA, how 
does Linux tell that there is a real OSA involved? My assumptions (which may be 
false ;-)) were that Linux as a z/VM guest should not be able to tell whether a 
NIC is real or virtual. And in our case the NIC is always virtual, because we 
do not connect directly to the OSA, we go through a vswitch. The bridging works 
as long as the traffic does not go through the OSA card, so somehow the Linux 
guest is able to pass frames with MAC addresses it does not own. These frames 
are only dropped if they go towards the OSA. I was not able to tell whether 
they are dropped by the OSA or by the vswitch which connects to the OSA. But 
the same vswitch passes the bridged packets with no problems to other Linux 
machines inside the box...

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

Thanks, this means a big change to our plans ;-) Do you know if there are any 
public docs (or source code) that we could look at to understand how the 
negotiation works?

Tomas

-Original Message-
From: Linux on 390 Port [mailto:LINUX-390@VM.MARIST.EDU] On Behalf Of Carsten 
Otte
Sent: Thursday, February 20, 2014 9:35 AM
To: LINUX-390@VM.MARIST.EDU
Subject: Re: Layer 2 frames passing through a Linux bridge get dropped before 
leaving the mainframe box

This setup won't work, because Linux negotiates its mac address with the OSA, 
and cannot send frames from another mac. You could use ip forwarding, and have 
Linux route on layer 3. This should work, as long as you use the OSA in layer 2 
mode.

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/

Re: Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Carsten Otte]

This setup won't work, because Linux negotiates its mac address with the
OSA, and cannot send frames from another mac. You could use ip forwarding,
and have Linux route on layer 3. This should work, as long as you use the
OSA
in layer 2 mode.

with kind regards
Carsten Otte
System z firmware development / Boeblingen lab
---
Every revolution was first a thought in one man's mind;
and when the same thought occurs to another man, it is the key to that era.

 - Ralph Waldo Emerson, Essays: First Series, 1841



 "Pavelka, Tomas"
   To
 Sent by: Linux on LINUX-390@vm.marist.edu,
 390 Port   cc
   Subject
   Layer 2 frames passing through a
   Linux bridge get dropped before
 20.02.2014 09:04  leaving the mainframe box


 Please respond to
 Linux on 390 Port
 






We have a problem where frames that pass through a Linux bridge do not
reach the gateway outside of the mainframe box. We have set up an
experiment that reproduces the problem, which looks like this:

(LINUX1) -  - (LINUXBR) -  - OSA - gateway

The problem is that in this setup we cannot ping the gateway. But, under a
different setup:

(LINUX1) -  - (LINUXBR) -  - (LINUX2)

Both LINUX1 and LINUX2 can communicate. Moreover, LINUX2 can ping the
gateway (the OSA card is still connected to the public vswitch, I just did
not put it in the picture).

Some more details that may be important:
- Both public and private vswitch are layer 2
- LINUXBR runs RHEL 6 and uses bridge-utils to create the bridge
- private vswitch is not connected to any OSA card

We have played with TCPDUMP and found that ARP (broadcast) packets do reach
the gateway and come back, but ping's ICMP (unicast) packets get dropped.
This led us to the following hypothesis: If there is a unicast packet
originating from a MAC address not known to public vswitch, it gets dropped
somewhere on the way between LINUXBR and the gateway.

Does anyone know any settings that could affect filtering done either by
the vswitch or by the OSA card? We asked our hardware people but they did
not know of anything that could cause the problems. But a more targeted
question could help if we knew what to ask for.

Any debugging tips will be much appreciated.

Thanks,
Tomas

Tomas Pavelka
CA Technologies
Sr Software Engineer
Tel:  +420226207796
tomas.pave...@ca.com

[cid:image001.gif@01CF2E1A.CF9FFDB0]<
http://www.ca.com/>

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
(Embedded image moved to file: pic04524.gif)

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/<>

Layer 2 frames passing through a Linux bridge get dropped before leaving the mainframe box

[Pavelka, Tomas]

We have a problem where frames that pass through a Linux bridge do not reach 
the gateway outside of the mainframe box. We have set up an experiment that 
reproduces the problem, which looks like this:

(LINUX1) -  - (LINUXBR) -  - OSA - gateway

The problem is that in this setup we cannot ping the gateway. But, under a 
different setup:

(LINUX1) -  - (LINUXBR) -  - (LINUX2)

Both LINUX1 and LINUX2 can communicate. Moreover, LINUX2 can ping the gateway 
(the OSA card is still connected to the public vswitch, I just did not put it 
in the picture).

Some more details that may be important:
- Both public and private vswitch are layer 2
- LINUXBR runs RHEL 6 and uses bridge-utils to create the bridge
- private vswitch is not connected to any OSA card

We have played with TCPDUMP and found that ARP (broadcast) packets do reach the 
gateway and come back, but ping's ICMP (unicast) packets get dropped. This led 
us to the following hypothesis: If there is a unicast packet originating from a 
MAC address not known to public vswitch, it gets dropped somewhere on the way 
between LINUXBR and the gateway.

Does anyone know any settings that could affect filtering done either by the 
vswitch or by the OSA card? We asked our hardware people but they did not know 
of anything that could cause the problems. But a more targeted question could 
help if we knew what to ask for.

Any debugging tips will be much appreciated.

Thanks,
Tomas

Tomas Pavelka
CA Technologies
Sr Software Engineer
Tel:  +420226207796
tomas.pave...@ca.com

[cid:image001.gif@01CF2E1A.CF9FFDB0]

--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to lists...@vm.marist.edu with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
<>