Re: z/VM, NTP, and the z/10.
The Linux ETR support only applies to Linux running in an LPAR. Getting this to work for Linuxrunning in virtual machines will require VM to accept and virtualize the new ETR signals. Rick Barlow Sr. z/VM Systems Programmer Nationwide Services Co., Technology Solutions, z/VM and System z Linux Support One Nationwide Plaza MB-02-201 Columbus OH 43215-2220 U.S.A Voice: (614) 249-5213Fax: (614) 249-3912 mailto:[EMAIL PROTECTED] The IBM z/VM Operating System IBMVM@LISTSERV.UARK.EDU wrote on 05/21/2008 05:08:14 PM: IBMVM@LISTSERV.UARK.EDU We've been running NTP for every with no ill effects. Our authentication product requires it be there (to active directory so presumably Kerberos is the reason). Now.. Funny this topic should come up. I was about to ask about this which came out the SLES 10 SP2 Release notes: http://www.novell.com/documentation/sles10/pdfdoc/release-notes_sp2/rele ase-notes_sp2.pdf * zSeries: ETR (external time reference) Support: Enables Linux images to synchronize with parallel Sysplex or GDPS by providing z/OS compatible external timer reference and maintaining data consistency groups for XRC data mover. More info: The ETR support introduces a new kernel parameter etr that is used to set the initial state for the online attribute of the two ports. The syntax of the parameter is etr=[on | off | port0 | port1]. The default is off. The ETR support introduces a number of sysfs attributes. There are two time synchronization ports etr0 and etr1, which can be accessed via sysfs under: /sys/devices/system/etr Is this for under z/VM as well ? (we are going to be doing XRC r.s.n.).
Re: z/VM, NTP, and the z/10.
On Wed, 21 May 2008 19:30:10 -0400, Alan Altmark [EMAIL PROTECTED] wrote: On Wednesday, 05/21/2008 at 07:25 EDT, dave [EMAIL PROTECTED] wrote: To recap: 1) VM (actually, CP) does not participate in the new SNT, once CP has it's hardware TOD clock set from either the HMC or the Operator, that's it, no changes to the h/w TOD clock. Disagree. See my previous posts. OK, I worded it poorlyI meant to say that there are no changes to the h/w clock that CP notices, even though STP is (slowly) adjusting the h/w clock to keep the VM LPAR in sync with other LPARs. (Hey, two out of three isn't bad for a Wednesday:-) DJ
Re: z/VM, NTP, and the z/10.
I don't think that IBM has announced the ability for z/VM to update its clock on the fly with input from an STP/NTP service. SLES10, however, can use an external NTP source to update its own clock (separate from the z/V M supervisor). I do this because our hobbit server will complain (softly bu t complain) when my z/VM based linux systems are 3 seconds different from t he network source that the hobbit server got its time from. /Tom Kern On Wed, 21 May 2008 11:44:28 -0500, Dave Jones [EMAIL PROTECTED] wrote: A client has just installed a z10 system, with the STP server and NTP cl ient support enabled. In one LPAR there is z/OS 1.9 running and exploiting an external time reference to keep it's LPAR time accurate. With the new z/ 10 STP and NTP client functions available, can the time in the z/VM LPAR al so be kept in sync with the time in the z/OS LPAR? If so, could this also b e used to keep Linux guests' time in sync as well? Many thanks for your *time* in help me understand this:-) DJ
Re: z/VM, NTP, and the z/10.
Thanks, Tom, I appreciate it. However, I would prefer not to have to run NTP clients in each of the Linux images due to the overhead that can produce. I think the point of confusion here is, at least to me, that the z10 technical overview document seems to imply that STP will be used to synchronize the time across LPARs and that PR/SM will (gradually) update the hardware TOD clock in each LAPR of a system and across systems, so that all of the hardware clocks are kept in sync. My question is: what happens if z/VM is running on one of those LPARs and PR/SM, under the covers, keeps updating z/VM's hardware TOD clock? Or am I completely missing the point here about what STP and NTP client features are supposed to provide? DJ - Original Message Follows - From: Thomas Kern [EMAIL PROTECTED] To: IBMVM@LISTSERV.UARK.EDU Subject: Re: z/VM, NTP, and the z/10. Date: Wed, 21 May 2008 11:49:55 -0500 I don't think that IBM has announced the ability for z/VM to update its clock on the fly with input from an STP/NTP service. SLES10, however, can use an external NTP source to update its own clock (separate from the z/VM supervisor). I do this because our hobbit server will complain (softly but complain) when my z/VM based linux systems are 3 seconds different from the network source that the hobbit server got its time from. /Tom Kern On Wed, 21 May 2008 11:44:28 -0500, Dave Jones [EMAIL PROTECTED] wrote: A client has just installed a z10 system, with the STP server and NTP client support enabled. In one LPAR there is z/OS 1.9 running and exploiting an external time reference to keep it's LPAR time accurate. With the new z/10 STP and NTP client functions available, can the time in the z/VM LPAR also be kept in sync with the time in the z/OS LPAR? If so, could this also be used to keep Linux guests' time in sync as well? Many thanks for your *time* in help me understand this:-) DJ
Re: z/VM, NTP, and the z/10.
To your first paragraph, Dave, actually, umm, no. The overhead is really small for [X]NTPD on Linux. I agree that NTP in each Linux is silly. But the evidence suggests that NTPD is the best behaved, smallest footprint, and all decency of the many daemons possessing virtual Linuxen. In my shop, XNTPD was the first background process we whacked. But it became clear that it was the *last* one we really needed to whack. We turned it back on. Running the NTP server is a whole lot better than even a daily 'ntpdate' via CRON. And precise time synch is vital to any multi-node transactional stuff. (NFS is a common example. Dunno how sensitive WAS and the like are to clock skew.) Now ... an 'ntpdate' at guest boot-up would suffice for local apps. (The clock is just that stable.) But we have lots of network relationships to worry about. The spiffy thing about time on System z is that the clock is incredibly stable. (Ticks at the right rate, though may be off by several minutes. It's always off by the same several minutes to great precision.) If we could get the underlying clock set really accurately, I suspect we'd still want to run NTP on most guests for validation and consistency. And incidentally, this is a problem in the VMware world too. (Unless they fixed it quite recently.) On Wed, May 21, 2008 at 1:41 PM, dave [EMAIL PROTECTED] wrote: Thanks, Tom, I appreciate it. However, I would prefer not to have to run NTP clients in each of the Linux images due to the overhead that can produce. I think the point of confusion here is, at least to me, that the z10 technical overview document seems to imply that STP will be used to synchronize the time across LPARs and that PR/SM will (gradually) update the hardware TOD clock in each LAPR of a system and across systems, so that all of the hardware clocks are kept in sync. My question is: what happens if z/VM is running on one of those LPARs and PR/SM, under the covers, keeps updating z/VM's hardware TOD clock? Or am I completely missing the point here about what STP and NTP client features are supposed to provide? DJ - Original Message Follows - From: Thomas Kern [EMAIL PROTECTED] To: IBMVM@LISTSERV.UARK.EDU Subject: Re: z/VM, NTP, and the z/10. Date: Wed, 21 May 2008 11:49:55 -0500 I don't think that IBM has announced the ability for z/VM to update its clock on the fly with input from an STP/NTP service. SLES10, however, can use an external NTP source to update its own clock (separate from the z/VM supervisor). I do this because our hobbit server will complain (softly but complain) when my z/VM based linux systems are 3 seconds different from the network source that the hobbit server got its time from. /Tom Kern On Wed, 21 May 2008 11:44:28 -0500, Dave Jones [EMAIL PROTECTED] wrote: A client has just installed a z10 system, with the STP server and NTP client support enabled. In one LPAR there is z/OS 1.9 running and exploiting an external time reference to keep it's LPAR time accurate. With the new z/10 STP and NTP client functions available, can the time in the z/VM LPAR also be kept in sync with the time in the z/OS LPAR? If so, could this also be used to keep Linux guests' time in sync as well? Many thanks for your *time* in help me understand this:-) DJ -- -- R;
Re: z/VM, NTP, and the z/10.
Rick, Now I'm confused. You write: Running the NTP server is a whole lot better than even a daily 'ntpdate' via CRON. and The spiffy thing about time on System z is that the clock is incredibly stable. So which is it? We wrote about having one server running the full xntpd to a reliable time source. I guess that would make it a stratum n-1 server. Then the other penguins in the colony run ntpdate nightly against the first server (thanks to Rob van der Heij for that suggestion). That would make them stratum n-2 servers. It seemed like a good balance. But we never tested it with an app that demands clock synchronization. So did you find that this model didn't work? Mike MacIsaac [EMAIL PROTECTED] (845) 433-7061
Re: z/VM, NTP, and the z/10.
Now I'm confused. You write: Running the NTP server is a whole lot better than even a daily 'ntpdate' via CRON. and The spiffy thing about time on System z is that the clock is incredibly stable. So which is it? Both are true - the key problem is that if the operator is off when he sets the time at VM startup, then all the clocks in the guests are off by the same amount, and if those clocks are involved in transactions spanning multiple LPARs, you'll see failures in authentication or transaction validation even though the hardware clock is keeping good solid time; it's keeping time based on a wrong starting point. VM never changes the clock after startup, so you can't bring the clocks into sync with the rest of the world without running xntpd in each individual guest. That said, xntpd is small, lightweight and well-behaved, so as not to add more entropy to the system it's running on. If you have to run a daemon in each guests, it's a polite and well-behaved one. We wrote about having one server running the full xntpd to a reliable time source. I guess that would make it a stratum n-1 server. Then the other penguins in the colony run ntpdate nightly against the first server (thanks to Rob van der Heij for that suggestion). That would make them stratum n-2 servers. It seemed like a good balance. But we never tested it with an app that demands clock synchronization. So did you find that this model didn't work? It doesn't work in that you still have to wake up every single guest and process something fairly often to keep the clocks synced, which generates paging and consumes CPU for no real good purpose. If VM were able to do it in CP (or get the updates when the other LPARs do), then we could probably rely on the clock in VM instead of individual software clocks, and dispense with xntpd in the guests. As Kerberos penetrates further into enterprises, this is going to get REALLY important. Kerberos is very sensitive to time sync in order to prevent replay attacks. Ditto Web Services-based apps - SOA really REALLY cares about this.
Re: z/VM, NTP, and the z/10.
The overhead isn't too much. We have a script in /etc/cron.daily that run s the ntpd program once and goes away. If I was concerned about the overhea d, I would do this at IPL and once a week since we don't drift more than a second every year, but our HMC clock is off by 3 seconds. I got the scrip t from one of the redbooks but can't remember which one. /Tom Kern On Wed, 21 May 2008 11:41:45 -0600, dave [EMAIL PROTECTED] wrote: Thanks, Tom, I appreciate it. However, I would prefer not to have to run NTP clients in each of the Linux images due to the overhead that can produce. I think the point of confusion here is, at least to me, that the z10 technical overview document seems to imply that STP will be used to synchronize the time across LPARs and that PR/SM will (gradually) update the hardware TOD clock in each LAPR of a system and across systems, so that all of the hardware clocks are kept in sync. My question is: what happens if z/VM is running on one of those LPARs and PR/SM, under the covers, keeps updating z/VM's hardware TOD clock? Or am I completely missing the point here about what STP and NTP client features are supposed to provide? DJ
Re: z/VM, NTP, and the z/10.
On Wednesday, 05/21/2008 at 12:45 EDT, Dave Jones [EMAIL PROTECTED] wrote: A client has just installed a z10 system, with the STP server and NTP client support enabled. In one LPAR there is z/OS 1.9 running and exploiting an external time reference to keep it's LPAR time accurate. With STP, you don't need an NTP client running in z/OS. z/OS will see the temporal displacement alerts generated by STP. The time shifts generated by STP are triggered by the NTP or other external time reference support. With the new z/10 STP and NTP client functions available, can the time in the z/VM LPAR also be kept in sync with the time in the z/OS LPAR? If so, could this also be used to keep Linux guests' time in sync as well? z/VM (CP) does not support STP and does not virtualize it. The fact that STP is getting its time via NTP doesn't matter. When the LPAR is activated, the current time will be given to the LPAR and the drift begins, with CP and all guests drifting together. But that's not as bad as it might seem. It has been true for some time that small changes in time due to normal clock drift have been dealt with by adjustments to the LPAR TOD clock with no STP or Sysplex Timer support required, making the TOD appear to advance at a slower or faster rate, as required to allow the TOD to drift back into sync again. This is called TOD steering. Only when the operator sets the time or when the NTP/ETR connection to the external time reference has been disconnected for a significant period of time (days/weeks) does the clock drifted sufficiently to require STP/Sysplex Timer support in the operating system. Alan Altmark z/VM Development IBM Endicott
Re: z/VM, NTP, and the z/10.
On Wed, May 21, 2008 at 8:53 PM, Richard Troth [EMAIL PROTECTED] wrote: The spiffy thing about time on System z is that the clock is incredibly stable. (Ticks at the right rate, though may be off by several minutes. It's always off by the same several minutes to great precision.) If we could get the underlying clock set really accurately, I suspect we'd still want to run NTP on most guests for validation and consistency. Sir Santa is very right about the stable System z clock. I measured this on old gear a few years ago, but have no reason to assume IBM made it worse. http://rvdheij.nl/Presentations/2005-L76.pdf So yes, the concern is where the hardware is very constant but different from all the remote systems (that often use NTP). This may bother some installations and applications. With Linux on z/VM, I think it is in general *not* a good idea to try to correct this with NTP. One of the problems is that CPU virtualisation. This is observed by Linux as small time variations. However, the nature of these variations is different from the type of clock defects that the NTP logic tries to compensate. So you end up with larger variations in an attempt to steer the clock. And there's a few more issues with the tools it that actually make it a lot worse, like stepping the clock backwards. I find it hard to believe that systems would have extreme time requirements but yet tolerate yanking the clock back The best solution is to synchronize your hardware TOD (at POR) with a reliable time source (i.e. not the $5 Mickey Mouse watch of the operator). In the old days we had the 9037-2 that would do that, and also steer the TOD carefully based on a weekly dial-up to some time service. But STP makes that a lot easier. So the best approach is to make z/VM run on time and let all virtual machines run at exact that same clock, so not running NTP themselves. If you let STP steer the hardware TOD then your systems continue to run on time. The acceleration of the TOD to catch up is so small that I can not imagine anything with Linux on z/VM notice it. It avoids the need for NTP completely, and provides a much more stable clock. If you don't have the gear to do this, then I would prefer a single virtual machine with to start up first and use NTP to obtain the right time and then uses SET VTOD according to that. The Linux virtual machines starting after that would ask CP to have the same time as that one and no need to run NTP either. And yes, running ntpdate now and then works, as long as your system can afford the clock being yanked (fortunately, all clocks I have seen run a weeny bit too slow, so ntpdate will step it forward, not back). Rob -- Rob van der Heij Velocity Software GmbH http://velocitysoftware.com/
Re: z/VM, NTP, and the z/10.
We've been running NTP for every with no ill effects. Our authentication product requires it be there (to active directory so presumably Kerberos is the reason). Now.. Funny this topic should come up. I was about to ask about this which came out the SLES 10 SP2 Release notes: http://www.novell.com/documentation/sles10/pdfdoc/release-notes_sp2/rele ase-notes_sp2.pdf * zSeries: ETR (external time reference) Support: Enables Linux images to synchronize with parallel Sysplex or GDPS by providing z/OS compatible external timer reference and maintaining data consistency groups for XRC data mover. More info: The ETR support introduces a new kernel parameter etr that is used to set the initial state for the online attribute of the two ports. The syntax of the parameter is etr=[on | off | port0 | port1]. The default is off. The ETR support introduces a number of sysfs attributes. There are two time synchronization ports etr0 and etr1, which can be accessed via sysfs under: /sys/devices/system/etr Is this for under z/VM as well ? (we are going to be doing XRC r.s.n.). Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
Re: z/VM, NTP, and the z/10.
Hi, Mike, ... Two things seem to have gotten run together in my post. I meant to say that running the NTP server on all guests is better in terms of impact to the VM host than running a poorly scheduled MULTIPLICITY of 'ntpdate' jobs nightly. At my shop, we introduced an arbitrary staggering of the guests. We have abandonned that. We again run the standard XNTPD on each guest. Once the Linux adjustment is established, the System z clock is so rock solid that the offset does not have to be revisitted often. Even if there is substantial drift, the NTP daemon appears to be much better behaved than other services. So did you find that this model didn't work? It's not that it doesn't work, it's just that we found the cost of changing from the Linux norm to be greater than the advantage. -- R;
Re: z/VM, NTP, and the z/10.
On Wednesday, 05/21/2008 at 02:43 EDT, dave [EMAIL PROTECTED] wrote: My question is: what happens if z/VM is running on one of those LPARs and PR/SM, under the covers, keeps updating z/VM's hardware TOD clock? As CP perceives time, nothing happens. The clock keeps on ticking. But to the outside observer the ticks will occur faster or slower, depending on whether the LPAR is ahead or behind. No sudden clock movements and no anti-time is introduced (time will not flow backwards). Think of it like slowly turning a screwdriver that adjusts the clock oscillator frequency. The drift is so small that it takes a long time (human scale) for the drift to reach the point at which steering is not possible. Only when the machine has no external time reference (e.g. NTP) will the clock drift that far. Only when the h/w is reconnected will STP tell the LPAR (if the opsys has asked to be told) that the time shift is too severe to steer back on course in a reasonable amount of time and that a Quantum Leap has happened. Alan Altmark z/VM Development IBM Endicott
Re: z/VM, NTP, and the z/10.
R; wrote: It's not that it doesn't work, it's just that we found the cost of changing from the Linux norm to be greater than the advantage. Exactly. There's enough other different areas that generate enough quizzical looks. Plus the unix security baseline here says it has to be run and I sure don't think the benefit is worth the effort of pursuing a security exception and all the fun that entails. Marcy This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
Re: z/VM, NTP, and the z/10.
As usual, the group comes through again; many thanks to all who have posted, it helped a lot. Yes, the problem the client is concerned about is implementing a security scheme across LPARs (and across CECs as well) that requires accurate time be kept. Not microsecond accurate, but more accurate that what the Operator can provide at VM IPL time from his watch. To recap: 1) VM (actually, CP) does not participate in the new SNT, once CP has it's hardware TOD clock set from either the HMC or the Operator, that's it, no changes to the h/w TOD clock. 2) It is feasible for Linux to run a network time protocol server as a guest of VM, so Linux images can keep their clocks in sync with an ETR. 3) Linux guests should *not* be enabled to get the CP timezone change external interrupt, they do that themselves. Does that pretty much sum up the consensus of the group? Many thanks again and have a good one. DJ - Original Message Follows - From: Alan Altmark [EMAIL PROTECTED] To: IBMVM@LISTSERV.UARK.EDU Subject: Re: z/VM, NTP, and the z/10. Date: Wed, 21 May 2008 19:08:05 -0400 On Wednesday, 05/21/2008 at 02:43 EDT, dave [EMAIL PROTECTED] wrote: My question is: what happens if z/VM is running on one of those LPARs and PR/SM, under the covers, keeps updating z/VM's hardware TOD clock? As CP perceives time, nothing happens. The clock keeps on ticking. But to the outside observer the ticks will occur faster or slower, depending on whether the LPAR is ahead or behind. No sudden clock movements and no anti-time is introduced (time will not flow backwards). Think of it like slowly turning a screwdriver that adjusts the clock oscillator frequency. The drift is so small that it takes a long time (human scale) for the drift to reach the point at which steering is not possible. Only when the machine has no external time reference (e.g. NTP) will the clock drift that far. Only when the h/w is reconnected will STP tell the LPAR (if the opsys has asked to be told) that the time shift is too severe to steer back on course in a reasonable amount of time and that a Quantum Leap has happened. Alan Altmark z/VM Development IBM Endicott
Re: z/VM, NTP, and the z/10.
On Wednesday, 05/21/2008 at 07:25 EDT, dave [EMAIL PROTECTED] wrote: To recap: 1) VM (actually, CP) does not participate in the new SNT, once CP has it's hardware TOD clock set from either the HMC or the Operator, that's it, no changes to the h/w TOD clock. Disagree. See my previous posts. 2) It is feasible for Linux to run a network time protocol server as a guest of VM, so Linux images can keep their clocks in sync with an ETR. Agreed. 3) Linux guests should *not* be enabled to get the CP timezone change external interrupt, they do that themselves. Agreed. Alan Altmark z/VM Development IBM Endicott
