Re: CTC Route Problem
Sometimes the hardest of problems have the simplest solutions. We ran some packet traces and confirmed that even with the CTCs out of the OSPF configuration they were still part of the source VIPA configuration. I really wanted to dedicate LPAR-LPAR traffic on the CTCs and leave the OSAs for normal network traffic. I am not going to mess with the qdio just yet because I must move this Linux LPAR to another processor in 2 months. I will revisit qdio at that time. For now, I simply removed the CTCs from the z/OS source VIPA configuration. This allowed me to dedicate the CTCs for high volume LPAR traffic without affecting the OSAs and network usage. I pass this along for those like myself that sometimes miss the boat because we take too long to stop and smell the flowers. Our VIPA configuration was: HOME xxx.xxx.98.20 MAINPEPT 10.28.93.20 FE1 ; OSA Express Fast Ethernet card 10.28.91.20 FE2 ; OSA Express Fast Ethernet card xxx.xxx.86.9CTCTP ;Test to Prod xxx.xxx.86.5CTCTC ;Test to CEC ; PRIMARYINTERFACE MAINPEPT We simply moved the CTCs before the primary interface like so to remove them from the source VIPA configuration: HOME xxx.xxx.86.5CTCTC ;Test to CEC xxx.xxx.86.9CTCTP ;Test to Prod 1xxx.xxx.98.20 MAINPEPT 10.28.93.20 FE1 ; OSA Express Fast Ethernet card 10.28.91.20 FE2 ; OSA Express Fast Ethernet card ; PRIMARYINTERFACE MAINPEPT Additional packet traces confirmed that the CTCs are no longer part of the source VIPA. This has been working great. Thanks you all that pointed me in the right direction. Peter -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
On Friday, 03/26/2004 at 06:24 EST, "Peter E. Abresch Jr. - at Pepco" <[EMAIL PROTECTED]> wrote: > I appreciate all your help and patience. There was/is still confusion on > my part. If it was easy, everyone could do it and the world wouldn't need me any more. (It kind of brings a tear to your eye, doesn't it?) > I thought that since I did not have the CTC from z/OS as part of > the OSPF routing and VIPA configuration that it would not contain the > source VIPA in the packet. I then figured that the source would be the > z/OS CTC. Since there was a static route to Linux from z/OS, the packet > should have no problem getting there. On Linux, there was another static > route to get it back. As you pointed out, I was wrong. Tricksy, it is. > Please bear with me > here though. I displayed my routes under Linux, and sure enough the z/OS > LPARs VIPA was learned correctly. See below. > > O>* 161.186.98.0/24 [110/11] via 10.28.91.20, eth1, 02:32:51 > via 10.28.93.20, eth0, 02:32:51 > O 161.186.98.20/32 [110/11] via 10.28.91.20, eth1, 02:32:51 > via 10.28.93.20, eth0, 02:32:51 > > The VIPA is being learned from the 2 OSA/e interfaces under z/OS as I > expected. > > However, these routes were not helping me. I had to insert the > following static to force the VIPA over the CTC instead of the OSA/e > interfaces. This is where my confusion begins again. If the interface a > ping request comes in on has no bearing on the interface chosen to return > the ping response, why was this static necessary? It looks like the packet > had a source VIPA of 161.186.98.20 but was hanging up in the network > somewhere. > > S>* 161.186.98.20/32 [1/0] is directly connected, ctc0 Without dynamic routing, there is no "learning". There are implicit routes based on the IP address and subnet mask. That's it. Anything beyond that must be explicitly coded by you. Remember, too, that VIPA is based on the illusion of routing. To Linux, a VIPA subnet or host is not directly attached (more easily seen on a network diagram). That means a routing entry is required. And you still want a default route defined. > I was hoping to reserve the CTC for high volume DB2 data and use the OSA/e > for normal network activity. A perfectly reasonable thing to do, though you might want to consider HiperSockets instead. Remember that the CTC is stuck at channel speed limits, no matter how fast the network or server. HiperSockets and OSA (QDIO) move data at CPU speed instead of channel speed. The faster the CPU, the faster the data movement. Alan Altmark Sr. Software Engineer IBM z/VM Development -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
I appreciate all your help and patience. There was/is still confusion on
my part. I thought that since I did not have the CTC from z/OS as part of
the OSPF routing and VIPA configuration that it would not contain the
source VIPA in the packet. I then figured that the source would be the
z/OS CTC. Since there was a static route to Linux from z/OS, the packet
should have no problem getting there. On Linux, there was another static
route to get it back. As you pointed out, I was wrong. Please bear with me
here though. I displayed my routes under Linux, and sure enough the z/OS
LPARs VIPA was learned correctly. See below.
O>* 161.186.98.0/24 [110/11] via 10.28.91.20, eth1, 02:32:51
via 10.28.93.20, eth0, 02:32:51
O 161.186.98.20/32 [110/11] via 10.28.91.20, eth1, 02:32:51
via 10.28.93.20, eth0, 02:32:51
The VIPA is being learned from the 2 OSA/e interfaces under z/OS as I
expected. However, these routes were not helping me. I had to insert the
following static to force the VIPA over the CTC instead of the OSA/e
interfaces. This is where my confusion begins again. If the interface a
ping request comes in on has no bearing on the interface chosen to return
the ping response, why was this static necessary? It looks like the packet
had a source VIPA of 161.186.98.20 but was hanging up in the network
somewhere.
S>* 161.186.98.20/32 [1/0] is directly connected, ctc0
I was hoping to reserve the CTC for high volume DB2 data and use the OSA/e
for normal network activity.
As far as Quagga, I will look at it this weekend. I am new to Linux and do
not fully understand how to install source RPMs yet.
Peter
Alan Altmark <[EMAIL PROTECTED]>
Sent by: Linux on 390 Port <[EMAIL PROTECTED]>
03/26/2004 02:38 PM
Please respond to Linux on 390 Port
To: [EMAIL PROTECTED]
cc:
Subject: Re: CTC Route Problem
On Friday, 03/26/2004 at 01:44 EST, "Peter E. Abresch Jr. - at Pepco"
<[EMAIL PROTECTED]> wrote:
> If I use the default route, I can ping. The default does not address the
> 2nd CTC from another z/OS LPAR. If also causes problems with users
coming
> from the OSA connections.
>
> My goal is to run VIPA on Linux using Zebra and OSPFD. This allows the
> Linux system to learn the default route. Of course this does not help us
> if the ping is not going back to the way it came.
>
> I should not need the default route. Linux knows about network
> 161.186.86.4/30 or as it stands now, 161.186.86.5/32. Why is Linux not
> sending the packet back via this network route? Why is it not going back
> the way it came? I am so confused. Maybe it is time for a beer :)
As I said before, those packets do not contain 161.186.86.5 as the origin
IP address. They contain the associated z/OS VIPA address (because you
have SOURCEVIPA active on z/OS).
Route selection is based entirely on the routing table. There is no
magic; if the Linux routing table does have an entry for your VIPA subnet
or host, or a default route, pointing Linux in the right direction, the
packet goes nowhere.
The interface a ping request comes in on has no bearing on the interface
chosen to return the ping response.
BTW, I suggest looking at Quagga. Zebra and OSPFD are dead. And if you
aren't actively running a routing daemon, you most certainly do need a
default gateway spec. Don't worry - when you start up the routing daemon,
it will override whatever you put in there.
Alan Altmark
Sr. Software Engineer
IBM z/VM Development
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or
visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
--
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
On Friday, 03/26/2004 at 01:44 EST, "Peter E. Abresch Jr. - at Pepco" <[EMAIL PROTECTED]> wrote: > If I use the default route, I can ping. The default does not address the > 2nd CTC from another z/OS LPAR. If also causes problems with users coming > from the OSA connections. > > My goal is to run VIPA on Linux using Zebra and OSPFD. This allows the > Linux system to learn the default route. Of course this does not help us > if the ping is not going back to the way it came. > > I should not need the default route. Linux knows about network > 161.186.86.4/30 or as it stands now, 161.186.86.5/32. Why is Linux not > sending the packet back via this network route? Why is it not going back > the way it came? I am so confused. Maybe it is time for a beer :) As I said before, those packets do not contain 161.186.86.5 as the origin IP address. They contain the associated z/OS VIPA address (because you have SOURCEVIPA active on z/OS). Route selection is based entirely on the routing table. There is no magic; if the Linux routing table does have an entry for your VIPA subnet or host, or a default route, pointing Linux in the right direction, the packet goes nowhere. The interface a ping request comes in on has no bearing on the interface chosen to return the ping response. BTW, I suggest looking at Quagga. Zebra and OSPFD are dead. And if you aren't actively running a routing daemon, you most certainly do need a default gateway spec. Don't worry - when you start up the routing daemon, it will override whatever you put in there. Alan Altmark Sr. Software Engineer IBM z/VM Development -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
More info: Pinging from z/OS is definitely traveling the CTC. Linux is not sending it back. I can ping the z/OS from Linux and it work great. I see the RX and TX increment with each ping. Now for the fun stuff. If I ping the z/OS VIPA from Linux, I can ping via the CTC from z/OS to Linux. Again I can see the RX and TX increment as expected. The inverse is also true. If I ping the Linux VIPA from z/OS, ping works fine. The pings will continue to work for about 5 minutes then they will stop working. If I ping the VIPAs again, I can get another 5 minutes. Does anyone have any ideas? That beer is looking better and better. Peter -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
If I use the default route, I can ping. The default does not address the 2nd CTC from another z/OS LPAR. If also causes problems with users coming from the OSA connections. My goal is to run VIPA on Linux using Zebra and OSPFD. This allows the Linux system to learn the default route. Of course this does not help us if the ping is not going back to the way it came. I should not need the default route. Linux knows about network 161.186.86.4/30 or as it stands now, 161.186.86.5/32. Why is Linux not sending the packet back via this network route? Why is it not going back the way it came? I am so confused. Maybe it is time for a beer :) Alan Altmark <[EMAIL PROTECTED]> Sent by: Linux on 390 Port <[EMAIL PROTECTED]> 03/26/2004 01:24 PM Please respond to Linux on 390 Port To: [EMAIL PROTECTED] cc: Subject: Re: CTC Route Problem On Friday, 03/26/2004 at 01:11 EST, "Peter E. Abresch Jr. - at Pepco" <[EMAIL PROTECTED]> wrote: > I made the change but still cannot ping from z/OS. Do you have a default route active? I noticed you have SOURCEVIPA active. That means the packets will have the VIPA address in them, not the CTC. Without a route entry, you're right, Linux won't know what to do. route add default gw 161.186.86.5 Alan Altmark Sr. Software Engineer IBM z/VM Development -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
On Friday, 03/26/2004 at 01:11 EST, "Peter E. Abresch Jr. - at Pepco" <[EMAIL PROTECTED]> wrote: > I made the change but still cannot ping from z/OS. Do you have a default route active? I noticed you have SOURCEVIPA active. That means the packets will have the VIPA address in them, not the CTC. Without a route entry, you're right, Linux won't know what to do. route add default gw 161.186.86.5 Alan Altmark Sr. Software Engineer IBM z/VM Development -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
I made the change but still cannot ping from z/OS. ctc0 Link encap:Serial Line IP inet addr:161.186.86.6 P-t-P:161.186.86.5 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MTU:32760 Metric:1 RX packets:63 errors:0 dropped:0 overruns:0 frame:0 TX packets:59 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:12120 (11.8 Kb) TX bytes:8374 (8.1 Kb) The CTC0 route changed to 161.186.86.5 dev ctc0 proto kernel scope link src 161.186.86.6 The ping is definitely making it over to Linux. It is just not coming back. Adam Thornton <[EMAIL PROTECTED]> Sent by: Linux on 390 Port <[EMAIL PROTECTED]> 03/26/2004 12:16 PM Please respond to Linux on 390 Port To: [EMAIL PROTECTED] cc: Subject: Re: CTC Route Problem On Fri, 2004-03-26 at 10:40, Peter E. Abresch Jr. - at Pepco wrote: > I am using CTCs between my Linux LPAR and two z/OS V1.2 LPARs. For > debugging, I shutdown Zebra, OSPFD, and all other interfaces with the > exception of 1 CTC. The CTC is as follows: > > Linux 161.186.86.6/30 --- z/OS V1.2 161.186.86.5/30 > > ctc0 Link encap:Serial Line IP > inet addr:161.186.86.6 P-t-P:161.186.86.5 Mask:255.255.255.252 I think this netmask should be 255.255.255.255, as the CTC is a point-to-point link. Adam -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
I have CTCs from the z/OS LPARs to the other z/OS LPARs. I have reviewed the z/OS routes but see no problems. The problem only exist between z/OS and Linux. It seems that Linux wants to send it somewhere else? We run OSPF on on z/OS with the CTC as static routes. He is an excerpt of the z/OS routes in case I missed something. ; DEVICE CTC2CTC 1062 IOBUFFERSIZE 32K AUTORESTART LINK CTCTC CTC 0 CTC2 ;; HOME 161.186.98.20 MAINPEPT 10.28.93.20 FE1 ; OSA Express Fast Ethernet card 10.28.91.20 FE2 ; OSA Express Fast Ethernet card 161.186.86.9CTCTP ;Test to Prod 161.186.86.5CTCTC ;Test to CEC ; IPCONFIG VARSUBNETTING IPCONFIG SOURCEVIPA IPCONFIG IGNOREREDIRECT IPCONFIG DEVRETRYDUR 0 IPCONFIG NODATAGRAMFWD ; ; ADD STATIC ROUTE FOR THE CTC ;HOST FIRST LINK PACKET ; HOP NAME SIZE GATEWAY ; ;NETWORK FIRST LINK PACKET SUBNET SUBNET ; HOP NAME SIZEMASK VALUE 161.186 = CTCTP65527 0.0.255.252 0.0.86.8 161.186 = CTCTC32760 0.0.255.252 0.0.86.4 ; PRIMARYINTERFACE MAINPEPT Adam Thornton <[EMAIL PROTECTED]> Sent by: Linux on 390 Port <[EMAIL PROTECTED]> 03/26/2004 12:25 PM Please respond to Linux on 390 Port To: [EMAIL PROTECTED] cc: Subject: Re: CTC Route Problem On Fri, 2004-03-26 at 10:40, Peter E. Abresch Jr. - at Pepco wrote: > Like I said, I can ping 161.186.86.5 from 161.186.86.6 with no problems. I > CANNOT ping 161.186.86.6 from 161.186.86.5. An ifconfig ctc0 shows that > the RX PACKETS increases by 1 each time I issue a ping from z/OS. This led > me to believe that the pings were getting to Linux from z/OS but not > finding their way back. I use static routes for the CTCs on both Linux and > z/OS. To simplify, I deleted all routes except for the CTC route which is > as follows: On z/OS, you must have an appropriate GATEWAY statement, saying that all traffic for 161.186.86.5 should go via CTC-whatever, or you wouldn't be able to ping Linux. > 161.186.86.4/30 dev ctc0 proto kernel scope link 161.186.86.6 > > I could not ping from z/OS. I added the following route and I can ping > Linux from z/OS via CTC all day long. > > 161.186.0.0/16 via 161.186.86.6 dec ctc0 scope link Try 161.186.86.6/32 via 161.186.86.5 and see what happens (that is, add a host route with no explicit dependence on the interface name), and after that try 0/0 via 161.186.86.6 to add a default route via z/OS. I'm always a little nervous about using device names rather than IP addresses to control routing. Adam -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
On Fri, 2004-03-26 at 10:40, Peter E. Abresch Jr. - at Pepco wrote: > Like I said, I can ping 161.186.86.5 from 161.186.86.6 with no problems. I > CANNOT ping 161.186.86.6 from 161.186.86.5. An ifconfig ctc0 shows that > the RX PACKETS increases by 1 each time I issue a ping from z/OS. This led > me to believe that the pings were getting to Linux from z/OS but not > finding their way back. I use static routes for the CTCs on both Linux and > z/OS. To simplify, I deleted all routes except for the CTC route which is > as follows: On z/OS, you must have an appropriate GATEWAY statement, saying that all traffic for 161.186.86.5 should go via CTC-whatever, or you wouldn't be able to ping Linux. > 161.186.86.4/30 dev ctc0 proto kernel scope link 161.186.86.6 > > I could not ping from z/OS. I added the following route and I can ping > Linux from z/OS via CTC all day long. > > 161.186.0.0/16 via 161.186.86.6 dec ctc0 scope link Try 161.186.86.6/32 via 161.186.86.5 and see what happens (that is, add a host route with no explicit dependence on the interface name), and after that try 0/0 via 161.186.86.6 to add a default route via z/OS. I'm always a little nervous about using device names rather than IP addresses to control routing. Adam -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: CTC Route Problem
On Fri, 2004-03-26 at 10:40, Peter E. Abresch Jr. - at Pepco wrote: > I am using CTCs between my Linux LPAR and two z/OS V1.2 LPARs. For > debugging, I shutdown Zebra, OSPFD, and all other interfaces with the > exception of 1 CTC. The CTC is as follows: > > Linux 161.186.86.6/30 --- z/OS V1.2 161.186.86.5/30 > > ctc0 Link encap:Serial Line IP > inet addr:161.186.86.6 P-t-P:161.186.86.5 Mask:255.255.255.252 I think this netmask should be 255.255.255.255, as the CTC is a point-to-point link. Adam -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
