Re: RSCS Connections
It doesn't quite fit what the network designers want. When the MV1N system, the one I simply called MVS1 in the original diagram, is available, they want all MVS1 traffic to go through it. The other problem is that during our attempts to get TCPNJE to work on MVS, we found that JES2 went crazy when it had either more than or less than 2 streams defined. I don't know if that was even reported to IBM because the MVS guys were satisfied that 2 streams worked. I guess that the best way for them to have automatic fail-over capabilities for the MVS1-plex is for them to build it into the network. Either that or have a group defined that contains the MVS1A, B, C, etc., machines defined but not the one normally handling the network for the plex. Then, if MV1N is down, they could route the traffic bound for MVS1 to the new group. It would take commands to switch, but it would work better than having all traffic just pile up until the network machine is available again. The business of the link is highly variable. Sometimes it sits idle for hours. Other times, it is busy enough to have as many as 200+ jobs pile up. Most jobs being submitted to MVS are small; some of the return traffic is quite significant (more than 10,000,000 records). Regards, Richard Schuh -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of David Boyes Sent: Sunday, April 26, 2009 7:44 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: RSCS Connections On 4/26/09 9:37 PM, Les Geer lg...@vnet.ibm.com wrote: I haven't looked through the code which handles how files are queued and transmitted in any great detail. However, what you describe sounds reasonable. In your MVSPLEX example, MVS1 must be first in the group list so it will always be picked. Or at least checked first, which should handle most of the cases. I think the check will count as a dispatch of the link task, which should cause the file to be processed there if there is an open stream on that link. Keep in mind the file is queued on each link. The first link to be free and dispatched will process the file. This may not always work as intended based on how busy the MVS1 link is. Yes. You will get an occasional job taking the direct path from RSCS to one of the execution nodes. If that's ok, cool. If not, then this probably doesn't do what you want. Another thing -- you'll probably want to set the JES resistance value very high for the direct RSCS-execution node links so that they are used outbound only as a last resort (ie when the normal link to MVS1 is not available).
Re: RSCS Connections
On Monday, 04/27/2009 at 12:12 EDT, Schuh, Richard rsc...@visa.com wrote: It doesn't quite fit what the network designers want. You *could* play the automation game. When MVSN goes down, RSCS will issue a message and try to restart the link. Capture the message and ROUTE the traffic to a backup node. When the MVSN link comes back up, reroute the traffic back to MVSN. Alan Altmark z/VM Development IBM Endicott
Re: RSCS Connections
That would work, but it would route the traffic during any period that MV1N was down. I suppose, we could start a timer in a service machine and have is issue the commands to switch to the group if the link was down and more than x jobs were queued or any job was queued for longer than some threshold value on the link. It would also have to be the one to detect that the link was back up and reroute the traffic back. Regards, Richard Schuh -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Alan Altmark Sent: Monday, April 27, 2009 9:16 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: RSCS Connections On Monday, 04/27/2009 at 12:12 EDT, Schuh, Richard rsc...@visa.com wrote: It doesn't quite fit what the network designers want. You *could* play the automation game. When MVSN goes down, RSCS will issue a message and try to restart the link. Capture the message and ROUTE the traffic to a backup node. When the MVSN link comes back up, reroute the traffic back to MVSN. Alan Altmark z/VM Development IBM Endicott
Re: RSCS Connections
On: Mon, Apr 27, 2009 at 09:27:14AM -0700,Schuh, Richard Wrote: } It would also have to be the one to detect that the link was back up and reroute the traffic back. One possible way of doing that is for the SVM to que an IEFBR14 type job to the link and check every n minutes to see if its gone. -- Rich Greenberg N Ft Myers, FL, USA richgr atsign panix.com + 1 239 543 1353 Eastern time. N6LRT I speak for myself my dogs only.VM'er since CP-67 Canines:Val, Red, Shasta Casey (RIP), Red Zero, Siberians Owner:Chinook-L Retired at the beach Asst Owner:Sibernet-L
Re: RSCS Connections
I don't suppose it's possible to duplicate the entry node MVS1N? If it's only role is to route files, it might be worth fixing the SPOF. Then you could have dual links and just let one go down with MVS1N. Another option would be an outboard NJE node that isn't dependent on MVS1N at all. Use an Intel box or a Linux guest as a shadow of the link between RSCS and MVS1N, and let the shadow box serve as the backup link. On 4/27/09 12:27 PM, Schuh, Richard rsc...@visa.com wrote: That would work, but it would route the traffic during any period that MV1N was down. I suppose, we could start a timer in a service machine and have is issue the commands to switch to the group if the link was down and more than x jobs were queued or any job was queued for longer than some threshold value on the link. It would also have to be the one to detect that the link was back up and reroute the traffic back. Regards, Richard Schuh
Re: RSCS Connections
The business of the link is highly variable. Sometimes it sits idle for hours. Other times, it is busy enough to have as many as 200+ jobs pile up. Most jobs being submitted to MVS are small; some of the return traffic is quite significant (more than 10,000,000 records). Routing that return traffic could be a major issue as well. Presumably, MVS1A/B/... are routing files back to your VM system via MVS1N. If it's down, then what? Mark Wheeler _ Windows Live™ SkyDrive™: Get 25 GB of free online storage. http://windowslive.com/online/skydrive?ocid=TXT_TAGLM_WL_skydrive_042009
Re: RSCS Connections
The best way is to write a Link State Change Exit to simply notify the SVM of any change to or from Connected. Regards, Richard Schuh -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Rich Greenberg Sent: Monday, April 27, 2009 9:37 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: RSCS Connections On: Mon, Apr 27, 2009 at 09:27:14AM -0700,Schuh, Richard Wrote: } It would also have to be the one to detect that the link was back up and reroute the traffic back. One possible way of doing that is for the SVM to que an IEFBR14 type job to the link and check every n minutes to see if its gone. -- Rich Greenberg N Ft Myers, FL, USA richgr atsign panix.com + 1 239 543 1353 Eastern time. N6LRT I speak for myself my dogs only. VM'er since CP-67 Canines:Val, Red, Shasta Casey (RIP), Red Zero, Siberians Owner:Chinook-L Retired at the beach Asst Owner:Sibernet-L
Re: RSCS Connections
Hmm. If it truly doesn't matter which node things run on, this might be worth trying: Define a link to MVS1 with 7 streams. Define links to each node of the plex with 1 stream each. Put all the links into a group called MVSPLEX. ROUTE (or REROUTE) MVS1 to GROUP MVSPLEX. What I think should happen is that the file will be queued onto the first link in the group that has an open stream, which will normally be MVS1, and you get the behavior you originally described. If all streams on MVS1 are busy, RSCS should select one of the other links and queue it directly to on e of the execution nodes that has a open stream for transmission. This should get you the behavior you want most of the time, and also remove the MVS1 front-end system as both a bottleneck and a single point of failur e in the configuration -- if link MVS1 fails, there are no streams available, and the jobs will (more slowly) get distributed to the execution systems directly. Admittedly, I haven't tried it, but it seems like it should work. Les, am I totally misunderstanding how link groups work? I haven't looked through the code which handles how files are queued and transmitted in any great detail. However, what you describe sounds reasonable. In your MVSPLEX example, MVS1 must be first in the group list so it will always be picked. Keep in mind the file is queued on each link. The first link to be free and dispatched will process the file. This may not always work as intended based on how busy the MVS1 link is. Best Regards, Les Geer IBM z/VM and Linux Development
Re: RSCS Connections
On 4/26/09 9:37 PM, Les Geer lg...@vnet.ibm.com wrote: I haven't looked through the code which handles how files are queued and transmitted in any great detail. However, what you describe sounds reasonable. In your MVSPLEX example, MVS1 must be first in the group list so it will always be picked. Or at least checked first, which should handle most of the cases. I think the check will count as a dispatch of the link task, which should cause the file to be processed there if there is an open stream on that link. Keep in mind the file is queued on each link. The first link to be free and dispatched will process the file. This may not always work as intended based on how busy the MVS1 link is. Yes. You will get an occasional job taking the direct path from RSCS to one of the execution nodes. If that's ok, cool. If not, then this probably doesn't do what you want. Another thing -- you'll probably want to set the JES resistance value very high for the direct RSCS-execution node links so that they are used outbound only as a last resort (ie when the normal link to MVS1 is not available).
Re: RSCS Connections
That is about what I thought. Sometimes you wish that you hadn't read things correctly. Regards, Richard Schuh -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Les Geer (607-429-3580) Sent: Thursday, April 23, 2009 7:00 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: RSCS Connections Following David Boyes' example= set up something like: LINKDEF MVS1A NODE MVS1 TYPE SNANJE LUNAME MVS1ALU ASTART RETRY PARM MVS1A BUFF=3D3840 LINKDEF MVS1B NODE MVS1 TYPE SNANJE LUNAME MVS1BLU ASTART RETRY PARM MVS1B BUFF=3D3840 ROUTE GROUP MVS1 TO MVS1A MVS1B I don't believe that will meet Richard's requirement. This defines MVS1 as a group of two nodes, MVS1A and MVS1B. All files for MVS1 will flow over these two links. Looking at the documentation, I don't see a good automated way to achieve Richard's goal. Best Regards, Les Geer IBM z/VM and Linux Development
Re: RSCS Connections
Clarification. What used to be MVS1 is now a front-end system that does not process the jobs addressed to MVS1. Instead, it distributes the work among two or more other machines, the ones I listed as MVS1A and MVS1B. There can, and probably will, be others in the future. The entire collection of machines behind the Front End appears as a single entity called MVS1 to RSCS. The only one that is currently connected to the NJE network is MVS1 (actually MVS1N - N for Network). Having set things up that way, the MVS JES2 guy is wondering if there was some way RSCS could use a back door to get jobs into at least one of the worker machines when MVS1N is down. They built a system having a single failure point and have experienced a failure. Now they are hoping for an easy, meaning VM, fix. The current return path is from the worker machines through MVS1N, which we know to be a choke point. All of the RSCS links to MVS systems (we have many more that the MVS1-plex) are TCPNJE. (Les can probably attest to the amount of effort that it took to get there. I hope that the JES2 development and support folks gave him a gold star for his help.) Regards, Richard Schuh From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Mark Wheeler Sent: Thursday, April 23, 2009 7:32 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: RSCS Connections Date: Thu, 23 Apr 2009 21:59:41 -0400 From: lg...@vnet.ibm.com Subject: Re: RSCS Connections To: IBMVM@LISTSERV.UARK.EDU Following David Boyes' example= set up something like: LINKDEF MVS1A NODE MVS1 TYPE SNANJE LUNAME MVS1ALU ASTART RETRY PARM MVS1A BUFF=3D3840 LINKDEF MVS1B NODE MVS1 TYPE SNANJE LUNAME MVS1BLU ASTART RETRY PARM MVS1B BUFF=3D3840 ROUTE GROUP MVS1 TO MVS1A MVS1B I don't believe that will meet Richard's requirement. This defines MVS1 as a group of two nodes, MVS1A and MVS1B. All files for MVS1 will flow over these two links. Looking at the documentation, I don't see a good automated way to achieve Richard's goal. Best Regards, Les Geer IBM z/VM and Linux Development I assumed (and we know what can happen when we assume) that MVS1(a) and MVS1(b) were in a shared MAS configuration. Richard will need to clarify. As an alternative, I wonder if DVIPA could be employed here? The RSCS system would use a TCPNJE link to MVS1, whose IP address could move between MVS1(a) and MVS1(b). I confess that I don't know if JES TCPNJE can exploit DVIPA. A reason I suspect that this must be a share MAS configuration is that files need to be able to flow from JES to RSCS from either MVS1(x) system. Again, need clarification. Mark Wheeler Windows Live(tm) Hotmail(r):...more than just e-mail. Check it out.http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_HM_more_042009
Re: RSCS Connections
I will ask our JES2 guy. There were APARS. What is your MVS/JES2 level? Regards, Richard Schuh -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Marcy Cortes Sent: Friday, April 24, 2009 9:34 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: RSCS Connections Richard wrote: All of the RSCS links to MVS systems (we have many more that the MVS1-plex) are TCPNJE. (Les can probably attest to the amount of effort that it took to get there. I hope that the JES2 development and support folks gave him a gold star for his help.) We may be going to TCPNJE now that it looks like NDM (Connect:Direct) doesn't require VTAM anymore. We can now shoot VTAM. Do you have this written up anywhere? Is there a list of z/OS apars or anything? Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
Re: RSCS Connections
I think they are in the process of rolling 1.10 to like 40 lpars. Thanks much! Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Schuh, Richard Sent: Friday, April 24, 2009 9:44 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: [IBMVM] RSCS Connections I will ask our JES2 guy. There were APARS. What is your MVS/JES2 level? Regards, Richard Schuh -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Marcy Cortes Sent: Friday, April 24, 2009 9:34 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: RSCS Connections Richard wrote: All of the RSCS links to MVS systems (we have many more that the MVS1-plex) are TCPNJE. (Les can probably attest to the amount of effort that it took to get there. I hope that the JES2 development and support folks gave him a gold star for his help.) We may be going to TCPNJE now that it looks like NDM (Connect:Direct) doesn't require VTAM anymore. We can now shoot VTAM. Do you have this written up anywhere? Is there a list of z/OS apars or anything? Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
Re: RSCS Connections
I set up z/OS 1.9 from the AD CD (June 2008) to talk from JES2 to my z/VM and (z and non-z) Linux systems using TCPNJE. There were no additional PTFs I needed to apply to make it work. On 4/24/09 12:53 PM, Marcy Cortes marcy.d.cor...@wellsfargo.com wrote: I think they are in the process of rolling 1.10 to like 40 lpars. Thanks much!
Re: RSCS Connections
Good to know! Thanks! Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of Neale Ferguson Sent: Friday, April 24, 2009 9:57 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: [IBMVM] RSCS Connections I set up z/OS 1.9 from the AD CD (June 2008) to talk from JES2 to my z/VM and (z and non-z) Linux systems using TCPNJE. There were no additional PTFs I needed to apply to make it work. On 4/24/09 12:53 PM, Marcy Cortes marcy.d.cor...@wellsfargo.com wrote: I think they are in the process of rolling 1.10 to like 40 lpars. Thanks much!
Re: RSCS Connections
On 4/23/09 8:00 PM, Mark Wheeler mwheele...@hotmail.com wrote: Following David Boyes' example, set up something like: LINKDEF MVS1A NODE MVS1 TYPE SNANJE LUNAME MVS1ALU ASTART RETRY PARM MVS1A BUFF=3840 LINKDEF MVS1B NODE MVS1 TYPE SNANJE LUNAME MVS1BLU ASTART RETRY PARM MVS1B BUFF=3840 ROUTE GROUP MVS1 TO MVS1A MVS1B Hmm. If it truly doesn't matter which node things run on, this might be worth trying: Define a link to MVS1 with 7 streams. Define links to each node of the plex with 1 stream each. Put all the links into a group called MVSPLEX. ROUTE (or REROUTE) MVS1 to GROUP MVSPLEX. What I think should happen is that the file will be queued onto the first link in the group that has an open stream, which will normally be MVS1, and you get the behavior you originally described. If all streams on MVS1 are busy, RSCS should select one of the other links and queue it directly to one of the execution nodes that has a open stream for transmission. This should get you the behavior you want most of the time, and also remove the MVS1 front-end system as both a bottleneck and a single point of failure in the configuration -- if link MVS1 fails, there are no streams available, and the jobs will (more slowly) get distributed to the execution systems directly. Admittedly, I haven't tried it, but it seems like it should work. Les, am I totally misunderstanding how link groups work? -- db
Re: RSCS Connections
I think you could use link groups here. You'd have to define MVS1A and MVS1B to VM, then create a group MVS1 that contains the MVS1A and MVS1B links. You could then route files passing through RSCS to GROUP MVS1 and it would pick the first available link to queue it on. If MVS1A went away, then traffic would flow via MVS1B until MVS1A came back, and would then go back to transmit on first-available.
Re: RSCS Connections
David, Our original idea was to have all jobs submitted to MVS1 and let the front end distribute the work. The users are unaware of the fact that their jobs run on different systems, the MVS1-plex is a single entity as far as they know. To make the GROUP work, it looks like I need to define the links for MVS1, MVS1A and MVS1B, then define the MVS1 Group with all three links. If the MVS1 link is chosen preferentially (order tested determined by position in the list?), then everything would be sent to MVS1, as desired. If it failed, then everything would be sent to either A or B, whichever responded first, but not both. Is this correct? Regards, Richard Schuh From: The IBM z/VM Operating System [mailto:ib...@listserv.uark.edu] On Behalf Of David Boyes Sent: Thursday, April 23, 2009 1:05 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: RSCS Connections I think you could use link groups here. You'd have to define MVS1A and MVS1B to VM, then create a group MVS1 that contains the MVS1A and MVS1B links. You could then route files passing through RSCS to GROUP MVS1 and it would pick the first available link to queue it on. If MVS1A went away, then traffic would flow via MVS1B until MVS1A came back, and would then go back to transmit on first-available.
Re: RSCS Connections
Richard, Following David Boyes' example, set up something like: LINKDEF MVS1A NODE MVS1 TYPE SNANJE LUNAME MVS1ALU ASTART RETRY PARM MVS1A BUFF=3840 LINKDEF MVS1B NODE MVS1 TYPE SNANJE LUNAME MVS1BLU ASTART RETRY PARM MVS1B BUFF=3840 ROUTE GROUP MVS1 TO MVS1A MVS1B Mark Wheeler http://www.linkedin.com/in/marklwheeler Date: Thu, 23 Apr 2009 13:00:45 -0700 From: rsc...@visa.com Subject: RSCS Connections To: IBMVM@LISTSERV.UARK.EDU We had a configuration that looks like this: VM/RSCS MVS1 The JES2 and MVS network folks got together and came up with a slight alteration: VM/RSCS - MVS1 | . Etc. | | MVS1(a) MVS1(b) Where VM knows absolutely nothing about the MVS1(x) subordinate machines. The node MVS1 has suddenly become a link to a group of systems collectively known as MVS1. Naturally, this means that if the Front End is down, the machines in the collection are unable to participate in the network. The question of keeping the MVS1 node alive when the F/E machine is down has been posed. The only way that I have been able to come up with so far is to define a link to MVS1A, for example, specify it as an alternate route for MVS1 traffic using something like route node mvs1 to link mvs1alternate mvs1a. If this will work, it will at least give access to one member of the collection. Will this work? If so, will the traffic revert to the normal path when the F/E is brought back up and signs in? Is there a better way? Regards, Richard Schuh _ Rediscover Hotmail®: Now available on your iPhone or BlackBerry http://windowslive.com/RediscoverHotmail?ocid=TXT_TAGLM_WL_HM_Rediscover_Mobile2_042009
Re: RSCS Connections
Following David Boyes' example= set up something like: LINKDEF MVS1A NODE MVS1 TYPE SNANJE LUNAME MVS1ALU ASTART RETRY PARM MVS1A BUFF=3D3840 LINKDEF MVS1B NODE MVS1 TYPE SNANJE LUNAME MVS1BLU ASTART RETRY PARM MVS1B BUFF=3D3840 ROUTE GROUP MVS1 TO MVS1A MVS1B I don't believe that will meet Richard's requirement. This defines MVS1 as a group of two nodes, MVS1A and MVS1B. All files for MVS1 will flow over these two links. Looking at the documentation, I don't see a good automated way to achieve Richard's goal. Best Regards, Les Geer IBM z/VM and Linux Development
Re: RSCS Connections
Date: Thu, 23 Apr 2009 21:59:41 -0400 From: lg...@vnet.ibm.com Subject: Re: RSCS Connections To: IBMVM@LISTSERV.UARK.EDU Following David Boyes' example= set up something like: LINKDEF MVS1A NODE MVS1 TYPE SNANJE LUNAME MVS1ALU ASTART RETRY PARM MVS1A BUFF=3D3840 LINKDEF MVS1B NODE MVS1 TYPE SNANJE LUNAME MVS1BLU ASTART RETRY PARM MVS1B BUFF=3D3840 ROUTE GROUP MVS1 TO MVS1A MVS1B I don't believe that will meet Richard's requirement. This defines MVS1 as a group of two nodes, MVS1A and MVS1B. All files for MVS1 will flow over these two links. Looking at the documentation, I don't see a good automated way to achieve Richard's goal. Best Regards, Les Geer IBM z/VM and Linux Development I assumed (and we know what can happen when we assume) that MVS1(a) and MVS1(b) were in a shared MAS configuration. Richard will need to clarify. As an alternative, I wonder if DVIPA could be employed here? The RSCS system would use a TCPNJE link to MVS1, whose IP address could move between MVS1(a) and MVS1(b). I confess that I don't know if JES TCPNJE can exploit DVIPA. A reason I suspect that this must be a share MAS configuration is that files need to be able to flow from JES to RSCS from either MVS1(x) system. Again, need clarification. Mark Wheeler _ Windows Live™ Hotmail®:…more than just e-mail. http://windowslive.com/online/hotmail?ocid=TXT_TAGLM_WL_HM_more_042009
