Re: [ACS41] System VMs not syncing time - does this block the release?
Isn't there some other way we can sync the time as a triage that would avoid a large test on the system vm template? Like adding a time flag to one of the commonly-run system vm scripts (or even just the ones that trigger tasks that require accurate time), which would run a 'date' command in the system vm, and modifying the applicable CitrixResourceBase methods to pass epoch timestamp along when these scripts are called? On Mon, May 20, 2013 at 2:05 PM, Chip Childers wrote: > We appear to be at an impasse here as well. I'm going to start a VOTE > on this issue, given that we have no easy answer. > > > On Wed, May 15, 2013 at 02:49:41PM -0700, Chiradeep Vittal wrote: >> Well, I disagree, from the perspective of hundreds of production clouds. >> No harm has been perceived in those clouds due to this defect. If they can >> live with it for several years, then they can live with it for a few more >> months. >> >> On 5/15/13 2:35 PM, "Wido den Hollander" wrote: >> >> > >> > >> >On 05/15/2013 11:33 PM, John Burwell wrote: >> >> Chiradeep, >> >> >> >> I disagree regarding the impact of this issue. Anyone running an SSVM >> >>will be affected by this issue because clocks will eventually drift >> >>(sooner rather than later) and when they do, any timestamps rendered by >> >>a system VM will unreliable (e.g. files creation and modified times, log >> >>entries, etc). When I put my sys admin hat on, that type behavior would >> >>4.1 an unacceptable release to me. >> >> >> > >> >Ack. Clocks should always be on time. No matter what a machine is doing. >> >Have the correct time is crucial. >> > >> >Think about log lines, debugging, etc, etc. >> > >> >Nothing wrong with running NTP in KVM System VM btw. >> > >> >Wido >> > >> >> Thanks, >> >> -John >> >> >> >> On May 15, 2013, at 5:24 PM, Chiradeep Vittal >> >> wrote: >> >> >> >>> Sure, I agree. But I'd also point out that for the vast majority of >> >>> CloudStack users (4.1 at least), this is not going to be an issue. I >> >>> suggest deferring this to 4.1.1 >> >>> A new template (easy or not) does require a full regression QA round. >> >>> >> >>> On 5/15/13 2:07 PM, "John Burwell" wrote: >> >>> >> Chiradeep, >> >> As I think thought it, I don't think a documentation note will >> sufficient >> because the SSVM can be destroyed and respawned by CloudStack without >> intervention by a human. Therefore, we can get into a situation >> where an >> operator installs and configures NTP, and then at some point in the >> future, the SSVM gets respawned and clocks drift. The worst part >> about >> this scenario is that the failures for S3 sync become silent since we >> have no mechanism to surface the failure to a dashboard or monitoring >> system. >> >> How much effort (i.e. hours/days) would it be to build a new image? >> >> Thanks, >> -John >> >> On May 15, 2013, at 4:59 PM, Chiradeep Vittal >> wrote: >> >> > Did some further digging around as to why >> > /proc/sys/xen/independent_wallclock is not there on the Debian system >> > vm. >> > >> > TLDR; the kernel is PvOps (I.e., just a regular kernel that works >> >like a >> > PV kernel, not a specialized paravirt kernel). To eliminate >> > special-casing, the independent_wallclock feature was dropped. >> >However, >> > this means that the domU clock is NO LONGER synch'ed to dom0 and NTP >> >is >> > required on ANY domU. So the clock on the domU is only sync'ed at >> >domU >> > creation time. I suspect Citrix XenServer might have some workaround >> > >> > >> > http://www.gossamer-threads.com/lists/xen/users/234750 >> > >> > >> > What this means is that we *may* need to run ntp on system vms on >> >Xen. >> > This is of course a non-trivial change involving updates to >> >systemvms. >> > >> > I suspect that it does not matter much for virtual routers or console >> > proxy vms. >> > >> > We could have an advisory that for those users that care (e.g., those >> > using S3 sync) that they need to run ntp after the SSVM has been >> > created. >> > I.e, login to the SSVM and run >> > apt-get update >> > apt-get install ntp >> > >> > -- >> > Chiradeep >> > >> > >> > >> > On 5/15/13 1:11 PM, "Chiradeep Vittal" >> > wrote: >> > >> >> For VMWare, the command >> >> vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit >> >>a >> >> patch for /etc/init.d/cloud-early-config to enable it >> >> >> >> >> >> On 5/15/13 12:23 PM, "Chiradeep Vittal" >> >> >> >> wrote: >> >> >> >>> I am not sure why it is missing, but I will refer to >> >>> Citrix XenServer 6.0 Virtual Machine Installation Guide >> >>> >> >>> http://s.apache.org/YGn >> >>> >> >>> And quote >> >>> "Time Handling in Linux VMs >>
Re: [ACS41] System VMs not syncing time - does this block the release?
We appear to be at an impasse here as well. I'm going to start a VOTE on this issue, given that we have no easy answer. On Wed, May 15, 2013 at 02:49:41PM -0700, Chiradeep Vittal wrote: > Well, I disagree, from the perspective of hundreds of production clouds. > No harm has been perceived in those clouds due to this defect. If they can > live with it for several years, then they can live with it for a few more > months. > > On 5/15/13 2:35 PM, "Wido den Hollander" wrote: > > > > > > >On 05/15/2013 11:33 PM, John Burwell wrote: > >> Chiradeep, > >> > >> I disagree regarding the impact of this issue. Anyone running an SSVM > >>will be affected by this issue because clocks will eventually drift > >>(sooner rather than later) and when they do, any timestamps rendered by > >>a system VM will unreliable (e.g. files creation and modified times, log > >>entries, etc). When I put my sys admin hat on, that type behavior would > >>4.1 an unacceptable release to me. > >> > > > >Ack. Clocks should always be on time. No matter what a machine is doing. > >Have the correct time is crucial. > > > >Think about log lines, debugging, etc, etc. > > > >Nothing wrong with running NTP in KVM System VM btw. > > > >Wido > > > >> Thanks, > >> -John > >> > >> On May 15, 2013, at 5:24 PM, Chiradeep Vittal > >> wrote: > >> > >>> Sure, I agree. But I'd also point out that for the vast majority of > >>> CloudStack users (4.1 at least), this is not going to be an issue. I > >>> suggest deferring this to 4.1.1 > >>> A new template (easy or not) does require a full regression QA round. > >>> > >>> On 5/15/13 2:07 PM, "John Burwell" wrote: > >>> > Chiradeep, > > As I think thought it, I don't think a documentation note will > sufficient > because the SSVM can be destroyed and respawned by CloudStack without > intervention by a human. Therefore, we can get into a situation > where an > operator installs and configures NTP, and then at some point in the > future, the SSVM gets respawned and clocks drift. The worst part > about > this scenario is that the failures for S3 sync become silent since we > have no mechanism to surface the failure to a dashboard or monitoring > system. > > How much effort (i.e. hours/days) would it be to build a new image? > > Thanks, > -John > > On May 15, 2013, at 4:59 PM, Chiradeep Vittal > wrote: > > > Did some further digging around as to why > > /proc/sys/xen/independent_wallclock is not there on the Debian system > > vm. > > > > TLDR; the kernel is PvOps (I.e., just a regular kernel that works > >like a > > PV kernel, not a specialized paravirt kernel). To eliminate > > special-casing, the independent_wallclock feature was dropped. > >However, > > this means that the domU clock is NO LONGER synch'ed to dom0 and NTP > >is > > required on ANY domU. So the clock on the domU is only sync'ed at > >domU > > creation time. I suspect Citrix XenServer might have some workaround > > > > > > http://www.gossamer-threads.com/lists/xen/users/234750 > > > > > > What this means is that we *may* need to run ntp on system vms on > >Xen. > > This is of course a non-trivial change involving updates to > >systemvms. > > > > I suspect that it does not matter much for virtual routers or console > > proxy vms. > > > > We could have an advisory that for those users that care (e.g., those > > using S3 sync) that they need to run ntp after the SSVM has been > > created. > > I.e, login to the SSVM and run > > apt-get update > > apt-get install ntp > > > > -- > > Chiradeep > > > > > > > > On 5/15/13 1:11 PM, "Chiradeep Vittal" > > wrote: > > > >> For VMWare, the command > >> vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit > >>a > >> patch for /etc/init.d/cloud-early-config to enable it > >> > >> > >> On 5/15/13 12:23 PM, "Chiradeep Vittal" > >> > >> wrote: > >> > >>> I am not sure why it is missing, but I will refer to > >>> Citrix XenServer 6.0 Virtual Machine Installation Guide > >>> > >>> http://s.apache.org/YGn > >>> > >>> And quote > >>> "Time Handling in Linux VMs > >>> By default, the clocks in a Linux VM are synchronized to the clock > >>> running > >>> on the control domain, and cannot be > >>> independently changed. This mode is a convenient default, since > >>>only > >>> the > >>> control domain needs to be running the > >>> NTP service to keep accurate time across all VMs. Upon installation > >>> of a > >>> new Linux VM, make sure you change the > >>> time-zone from the default UTC to your local value (see the section > >>> called > >>> ³Release Notes² for specific distribution > >>> instructions). >
Re: [ACS41] System VMs not syncing time - does this block the release?
On Mon, May 20, 2013 at 3:59 PM, Chip Childers wrote: > On Wed, May 15, 2013 at 01:11:53PM -0700, Chiradeep Vittal wrote: >> For VMWare, the command >> vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a >> patch for /etc/init.d/cloud-early-config to enable it > > Please do! > > -chip Ignore me. You already did.
Re: [ACS41] System VMs not syncing time - does this block the release?
On Wed, May 15, 2013 at 01:11:53PM -0700, Chiradeep Vittal wrote: > For VMWare, the command > vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a > patch for /etc/init.d/cloud-early-config to enable it Please do! -chip
Re: [ACS41] System VMs not syncing time - does this block the release?
> > Nothing wrong with running NTP in KVM System VM btw. > > Wido > I think they are right that it will be fighting with the hypervisor synced clock. It might not really pose much of an issue technically, though.
Re: [ACS41] System VMs not syncing time - does this block the release?
Well, I disagree, from the perspective of hundreds of production clouds. No harm has been perceived in those clouds due to this defect. If they can live with it for several years, then they can live with it for a few more months. On 5/15/13 2:35 PM, "Wido den Hollander" wrote: > > >On 05/15/2013 11:33 PM, John Burwell wrote: >> Chiradeep, >> >> I disagree regarding the impact of this issue. Anyone running an SSVM >>will be affected by this issue because clocks will eventually drift >>(sooner rather than later) and when they do, any timestamps rendered by >>a system VM will unreliable (e.g. files creation and modified times, log >>entries, etc). When I put my sys admin hat on, that type behavior would >>4.1 an unacceptable release to me. >> > >Ack. Clocks should always be on time. No matter what a machine is doing. >Have the correct time is crucial. > >Think about log lines, debugging, etc, etc. > >Nothing wrong with running NTP in KVM System VM btw. > >Wido > >> Thanks, >> -John >> >> On May 15, 2013, at 5:24 PM, Chiradeep Vittal >> wrote: >> >>> Sure, I agree. But I'd also point out that for the vast majority of >>> CloudStack users (4.1 at least), this is not going to be an issue. I >>> suggest deferring this to 4.1.1 >>> A new template (easy or not) does require a full regression QA round. >>> >>> On 5/15/13 2:07 PM, "John Burwell" wrote: >>> Chiradeep, As I think thought it, I don't think a documentation note will sufficient because the SSVM can be destroyed and respawned by CloudStack without intervention by a human. Therefore, we can get into a situation where an operator installs and configures NTP, and then at some point in the future, the SSVM gets respawned and clocks drift. The worst part about this scenario is that the failures for S3 sync become silent since we have no mechanism to surface the failure to a dashboard or monitoring system. How much effort (i.e. hours/days) would it be to build a new image? Thanks, -John On May 15, 2013, at 4:59 PM, Chiradeep Vittal wrote: > Did some further digging around as to why > /proc/sys/xen/independent_wallclock is not there on the Debian system > vm. > > TLDR; the kernel is PvOps (I.e., just a regular kernel that works >like a > PV kernel, not a specialized paravirt kernel). To eliminate > special-casing, the independent_wallclock feature was dropped. >However, > this means that the domU clock is NO LONGER synch'ed to dom0 and NTP >is > required on ANY domU. So the clock on the domU is only sync'ed at >domU > creation time. I suspect Citrix XenServer might have some workaround > > > http://www.gossamer-threads.com/lists/xen/users/234750 > > > What this means is that we *may* need to run ntp on system vms on >Xen. > This is of course a non-trivial change involving updates to >systemvms. > > I suspect that it does not matter much for virtual routers or console > proxy vms. > > We could have an advisory that for those users that care (e.g., those > using S3 sync) that they need to run ntp after the SSVM has been > created. > I.e, login to the SSVM and run > apt-get update > apt-get install ntp > > -- > Chiradeep > > > > On 5/15/13 1:11 PM, "Chiradeep Vittal" > wrote: > >> For VMWare, the command >> vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit >>a >> patch for /etc/init.d/cloud-early-config to enable it >> >> >> On 5/15/13 12:23 PM, "Chiradeep Vittal" >> >> wrote: >> >>> I am not sure why it is missing, but I will refer to >>> Citrix XenServer 6.0 Virtual Machine Installation Guide >>> >>> http://s.apache.org/YGn >>> >>> And quote >>> "Time Handling in Linux VMs >>> By default, the clocks in a Linux VM are synchronized to the clock >>> running >>> on the control domain, and cannot be >>> independently changed. This mode is a convenient default, since >>>only >>> the >>> control domain needs to be running the >>> NTP service to keep accurate time across all VMs. Upon installation >>> of a >>> new Linux VM, make sure you change the >>> time-zone from the default UTC to your local value (see the section >>> called >>> ³Release Notes² for specific distribution >>> instructions). >>> To set individual Linux VMs to maintain independent times >>> 1. From a root prompt on the VM, run the command: echo 1 > >>> /proc/sys/xen/independent_wallclock >>> 2. This can be persisted across reboots by changing the >>> /etc/sysctl.conf >>> configuration file and adding: >>> # Set independent wall clock time >>> xen.independent_wallclock=1 >>> 3. As a third alternative, independent_wallclock=1 may also be >>>pas
Re: [ACS41] System VMs not syncing time - does this block the release?
On 05/15/2013 11:33 PM, John Burwell wrote: > Chiradeep, > > I disagree regarding the impact of this issue. Anyone running an SSVM will > be affected by this issue because clocks will eventually drift (sooner rather > than later) and when they do, any timestamps rendered by a system VM will > unreliable (e.g. files creation and modified times, log entries, etc). When > I put my sys admin hat on, that type behavior would 4.1 an unacceptable > release to me. > Ack. Clocks should always be on time. No matter what a machine is doing. Have the correct time is crucial. Think about log lines, debugging, etc, etc. Nothing wrong with running NTP in KVM System VM btw. Wido > Thanks, > -John > > On May 15, 2013, at 5:24 PM, Chiradeep Vittal > wrote: > >> Sure, I agree. But I'd also point out that for the vast majority of >> CloudStack users (4.1 at least), this is not going to be an issue. I >> suggest deferring this to 4.1.1 >> A new template (easy or not) does require a full regression QA round. >> >> On 5/15/13 2:07 PM, "John Burwell" wrote: >> >>> Chiradeep, >>> >>> As I think thought it, I don't think a documentation note will sufficient >>> because the SSVM can be destroyed and respawned by CloudStack without >>> intervention by a human. Therefore, we can get into a situation where an >>> operator installs and configures NTP, and then at some point in the >>> future, the SSVM gets respawned and clocks drift. The worst part about >>> this scenario is that the failures for S3 sync become silent since we >>> have no mechanism to surface the failure to a dashboard or monitoring >>> system. >>> >>> How much effort (i.e. hours/days) would it be to build a new image? >>> >>> Thanks, >>> -John >>> >>> On May 15, 2013, at 4:59 PM, Chiradeep Vittal >>> wrote: >>> Did some further digging around as to why /proc/sys/xen/independent_wallclock is not there on the Debian system vm. TLDR; the kernel is PvOps (I.e., just a regular kernel that works like a PV kernel, not a specialized paravirt kernel). To eliminate special-casing, the independent_wallclock feature was dropped. However, this means that the domU clock is NO LONGER synch'ed to dom0 and NTP is required on ANY domU. So the clock on the domU is only sync'ed at domU creation time. I suspect Citrix XenServer might have some workaround http://www.gossamer-threads.com/lists/xen/users/234750 What this means is that we *may* need to run ntp on system vms on Xen. This is of course a non-trivial change involving updates to systemvms. I suspect that it does not matter much for virtual routers or console proxy vms. We could have an advisory that for those users that care (e.g., those using S3 sync) that they need to run ntp after the SSVM has been created. I.e, login to the SSVM and run apt-get update apt-get install ntp -- Chiradeep On 5/15/13 1:11 PM, "Chiradeep Vittal" wrote: > For VMWare, the command > vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a > patch for /etc/init.d/cloud-early-config to enable it > > > On 5/15/13 12:23 PM, "Chiradeep Vittal" > wrote: > >> I am not sure why it is missing, but I will refer to >> Citrix XenServer 6.0 Virtual Machine Installation Guide >> >> http://s.apache.org/YGn >> >> And quote >> "Time Handling in Linux VMs >> By default, the clocks in a Linux VM are synchronized to the clock >> running >> on the control domain, and cannot be >> independently changed. This mode is a convenient default, since only >> the >> control domain needs to be running the >> NTP service to keep accurate time across all VMs. Upon installation >> of a >> new Linux VM, make sure you change the >> time-zone from the default UTC to your local value (see the section >> called >> ³Release Notes² for specific distribution >> instructions). >> To set individual Linux VMs to maintain independent times >> 1. From a root prompt on the VM, run the command: echo 1 > >> /proc/sys/xen/independent_wallclock >> 2. This can be persisted across reboots by changing the >> /etc/sysctl.conf >> configuration file and adding: >> # Set independent wall clock time >> xen.independent_wallclock=1 >> 3. As a third alternative, independent_wallclock=1 may also be passed >> as >> a >> boot parameter to the VM. >> " >> >> >> On 5/15/13 12:09 PM, "Chip Childers" >> wrote: >> >>> On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: /proc/sys is not a regular filesystem and cannot be added to from the shell. Drivers need to add nodes into this filesystem. >>> >>> Backing up a bit... >>> >>> This is the current system VM im
Re: [ACS41] System VMs not syncing time - does this block the release?
Chiradeep, I disagree regarding the impact of this issue. Anyone running an SSVM will be affected by this issue because clocks will eventually drift (sooner rather than later) and when they do, any timestamps rendered by a system VM will unreliable (e.g. files creation and modified times, log entries, etc). When I put my sys admin hat on, that type behavior would 4.1 an unacceptable release to me. Thanks, -John On May 15, 2013, at 5:24 PM, Chiradeep Vittal wrote: > Sure, I agree. But I'd also point out that for the vast majority of > CloudStack users (4.1 at least), this is not going to be an issue. I > suggest deferring this to 4.1.1 > A new template (easy or not) does require a full regression QA round. > > On 5/15/13 2:07 PM, "John Burwell" wrote: > >> Chiradeep, >> >> As I think thought it, I don't think a documentation note will sufficient >> because the SSVM can be destroyed and respawned by CloudStack without >> intervention by a human. Therefore, we can get into a situation where an >> operator installs and configures NTP, and then at some point in the >> future, the SSVM gets respawned and clocks drift. The worst part about >> this scenario is that the failures for S3 sync become silent since we >> have no mechanism to surface the failure to a dashboard or monitoring >> system. >> >> How much effort (i.e. hours/days) would it be to build a new image? >> >> Thanks, >> -John >> >> On May 15, 2013, at 4:59 PM, Chiradeep Vittal >> wrote: >> >>> Did some further digging around as to why >>> /proc/sys/xen/independent_wallclock is not there on the Debian system >>> vm. >>> >>> TLDR; the kernel is PvOps (I.e., just a regular kernel that works like a >>> PV kernel, not a specialized paravirt kernel). To eliminate >>> special-casing, the independent_wallclock feature was dropped. However, >>> this means that the domU clock is NO LONGER synch'ed to dom0 and NTP is >>> required on ANY domU. So the clock on the domU is only sync'ed at domU >>> creation time. I suspect Citrix XenServer might have some workaround >>> >>> >>> http://www.gossamer-threads.com/lists/xen/users/234750 >>> >>> >>> What this means is that we *may* need to run ntp on system vms on Xen. >>> This is of course a non-trivial change involving updates to systemvms. >>> >>> I suspect that it does not matter much for virtual routers or console >>> proxy vms. >>> >>> We could have an advisory that for those users that care (e.g., those >>> using S3 sync) that they need to run ntp after the SSVM has been >>> created. >>> I.e, login to the SSVM and run >>> apt-get update >>> apt-get install ntp >>> >>> -- >>> Chiradeep >>> >>> >>> >>> On 5/15/13 1:11 PM, "Chiradeep Vittal" >>> wrote: >>> For VMWare, the command vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a patch for /etc/init.d/cloud-early-config to enable it On 5/15/13 12:23 PM, "Chiradeep Vittal" wrote: > I am not sure why it is missing, but I will refer to > Citrix XenServer 6.0 Virtual Machine Installation Guide > > http://s.apache.org/YGn > > And quote > "Time Handling in Linux VMs > By default, the clocks in a Linux VM are synchronized to the clock > running > on the control domain, and cannot be > independently changed. This mode is a convenient default, since only > the > control domain needs to be running the > NTP service to keep accurate time across all VMs. Upon installation > of a > new Linux VM, make sure you change the > time-zone from the default UTC to your local value (see the section > called > ³Release Notes² for specific distribution > instructions). > To set individual Linux VMs to maintain independent times > 1. From a root prompt on the VM, run the command: echo 1 > > /proc/sys/xen/independent_wallclock > 2. This can be persisted across reboots by changing the > /etc/sysctl.conf > configuration file and adding: > # Set independent wall clock time > xen.independent_wallclock=1 > 3. As a third alternative, independent_wallclock=1 may also be passed > as > a > boot parameter to the VM. > " > > > On 5/15/13 12:09 PM, "Chip Childers" > wrote: > >> On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: >>> /proc/sys is not a regular filesystem and cannot be added to from >>> the >>> shell. >>> Drivers need to add nodes into this filesystem. >> >> Backing up a bit... >> >> This is the current system VM image that the 4.1 software should be >> using on Xen: >> >> http://download.cloud.com/templates/acton/acton- >> systemvm-02062012.vhd.bz2 >> >> Does this system VM have the correct drives installed to set >> /proc/sys/xen to use the Dom0 time for sync? >> >> If so, then is this a devcloud only issue for Xen? If that's the >> case, >>>
Re: [ACS41] System VMs not syncing time - does this block the release?
Sure, I agree. But I'd also point out that for the vast majority of CloudStack users (4.1 at least), this is not going to be an issue. I suggest deferring this to 4.1.1 A new template (easy or not) does require a full regression QA round. On 5/15/13 2:07 PM, "John Burwell" wrote: >Chiradeep, > >As I think thought it, I don't think a documentation note will sufficient >because the SSVM can be destroyed and respawned by CloudStack without >intervention by a human. Therefore, we can get into a situation where an >operator installs and configures NTP, and then at some point in the >future, the SSVM gets respawned and clocks drift. The worst part about >this scenario is that the failures for S3 sync become silent since we >have no mechanism to surface the failure to a dashboard or monitoring >system. > >How much effort (i.e. hours/days) would it be to build a new image? > >Thanks, >-John > >On May 15, 2013, at 4:59 PM, Chiradeep Vittal > wrote: > >> Did some further digging around as to why >> /proc/sys/xen/independent_wallclock is not there on the Debian system >>vm. >> >> TLDR; the kernel is PvOps (I.e., just a regular kernel that works like a >> PV kernel, not a specialized paravirt kernel). To eliminate >> special-casing, the independent_wallclock feature was dropped. However, >> this means that the domU clock is NO LONGER synch'ed to dom0 and NTP is >> required on ANY domU. So the clock on the domU is only sync'ed at domU >> creation time. I suspect Citrix XenServer might have some workaround >> >> >> http://www.gossamer-threads.com/lists/xen/users/234750 >> >> >> What this means is that we *may* need to run ntp on system vms on Xen. >> This is of course a non-trivial change involving updates to systemvms. >> >> I suspect that it does not matter much for virtual routers or console >> proxy vms. >> >> We could have an advisory that for those users that care (e.g., those >> using S3 sync) that they need to run ntp after the SSVM has been >>created. >> I.e, login to the SSVM and run >> apt-get update >> apt-get install ntp >> >> -- >> Chiradeep >> >> >> >> On 5/15/13 1:11 PM, "Chiradeep Vittal" >>wrote: >> >>> For VMWare, the command >>> vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a >>> patch for /etc/init.d/cloud-early-config to enable it >>> >>> >>> On 5/15/13 12:23 PM, "Chiradeep Vittal" >>> wrote: >>> I am not sure why it is missing, but I will refer to Citrix XenServer 6.0 Virtual Machine Installation Guide http://s.apache.org/YGn And quote "Time Handling in Linux VMs By default, the clocks in a Linux VM are synchronized to the clock running on the control domain, and cannot be independently changed. This mode is a convenient default, since only the control domain needs to be running the NTP service to keep accurate time across all VMs. Upon installation of a new Linux VM, make sure you change the time-zone from the default UTC to your local value (see the section called ³Release Notes² for specific distribution instructions). To set individual Linux VMs to maintain independent times 1. From a root prompt on the VM, run the command: echo 1 > /proc/sys/xen/independent_wallclock 2. This can be persisted across reboots by changing the /etc/sysctl.conf configuration file and adding: # Set independent wall clock time xen.independent_wallclock=1 3. As a third alternative, independent_wallclock=1 may also be passed as a boot parameter to the VM. " On 5/15/13 12:09 PM, "Chip Childers" wrote: > On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: >> /proc/sys is not a regular filesystem and cannot be added to from >>the >> shell. >> Drivers need to add nodes into this filesystem. > > Backing up a bit... > > This is the current system VM image that the 4.1 software should be > using on Xen: > > http://download.cloud.com/templates/acton/acton- > systemvm-02062012.vhd.bz2 > > Does this system VM have the correct drives installed to set > /proc/sys/xen to use the Dom0 time for sync? > > If so, then is this a devcloud only issue for Xen? If that's the >case, > then we shouldn't block a release to simply improve things. > > We do know that a patch for kvm was needed. > > We still don't have any > answer or comments about the VMware system VM (specifically this > template: http://download.cloud.com/templates/burbank/burbank- > systemvm-08012012.ova > >>> >> >
Re: [ACS41] System VMs not syncing time - does this block the release?
All, One other point to consider is that we only want NTP running for Xen system VMs. Running NTP and VMWare Tools together causes big problems because the two fight each other to sync the clock. I don;t know about KVM, but I wouldn't be surprised if it doesn't have same types of problems. Thanks, -John On May 15, 2013, at 5:07 PM, John Burwell wrote: > Chiradeep, > > As I think thought it, I don't think a documentation note will sufficient > because the SSVM can be destroyed and respawned by CloudStack without > intervention by a human. Therefore, we can get into a situation where an > operator installs and configures NTP, and then at some point in the future, > the SSVM gets respawned and clocks drift. The worst part about this scenario > is that the failures for S3 sync become silent since we have no mechanism to > surface the failure to a dashboard or monitoring system. > > How much effort (i.e. hours/days) would it be to build a new image? > > Thanks, > -John > > On May 15, 2013, at 4:59 PM, Chiradeep Vittal > wrote: > >> Did some further digging around as to why >> /proc/sys/xen/independent_wallclock is not there on the Debian system vm. >> >> TLDR; the kernel is PvOps (I.e., just a regular kernel that works like a >> PV kernel, not a specialized paravirt kernel). To eliminate >> special-casing, the independent_wallclock feature was dropped. However, >> this means that the domU clock is NO LONGER synch'ed to dom0 and NTP is >> required on ANY domU. So the clock on the domU is only sync'ed at domU >> creation time. I suspect Citrix XenServer might have some workaround >> >> >> http://www.gossamer-threads.com/lists/xen/users/234750 >> >> >> What this means is that we *may* need to run ntp on system vms on Xen. >> This is of course a non-trivial change involving updates to systemvms. >> >> I suspect that it does not matter much for virtual routers or console >> proxy vms. >> >> We could have an advisory that for those users that care (e.g., those >> using S3 sync) that they need to run ntp after the SSVM has been created. >> I.e, login to the SSVM and run >> apt-get update >> apt-get install ntp >> >> -- >> Chiradeep >> >> >> >> On 5/15/13 1:11 PM, "Chiradeep Vittal" wrote: >> >>> For VMWare, the command >>> vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a >>> patch for /etc/init.d/cloud-early-config to enable it >>> >>> >>> On 5/15/13 12:23 PM, "Chiradeep Vittal" >>> wrote: >>> I am not sure why it is missing, but I will refer to Citrix XenServer 6.0 Virtual Machine Installation Guide http://s.apache.org/YGn And quote "Time Handling in Linux VMs By default, the clocks in a Linux VM are synchronized to the clock running on the control domain, and cannot be independently changed. This mode is a convenient default, since only the control domain needs to be running the NTP service to keep accurate time across all VMs. Upon installation of a new Linux VM, make sure you change the time-zone from the default UTC to your local value (see the section called ³Release Notes² for specific distribution instructions). To set individual Linux VMs to maintain independent times 1. From a root prompt on the VM, run the command: echo 1 > /proc/sys/xen/independent_wallclock 2. This can be persisted across reboots by changing the /etc/sysctl.conf configuration file and adding: # Set independent wall clock time xen.independent_wallclock=1 3. As a third alternative, independent_wallclock=1 may also be passed as a boot parameter to the VM. " On 5/15/13 12:09 PM, "Chip Childers" wrote: > On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: >> /proc/sys is not a regular filesystem and cannot be added to from the >> shell. >> Drivers need to add nodes into this filesystem. > > Backing up a bit... > > This is the current system VM image that the 4.1 software should be > using on Xen: > > http://download.cloud.com/templates/acton/acton- > systemvm-02062012.vhd.bz2 > > Does this system VM have the correct drives installed to set > /proc/sys/xen to use the Dom0 time for sync? > > If so, then is this a devcloud only issue for Xen? If that's the case, > then we shouldn't block a release to simply improve things. > > We do know that a patch for kvm was needed. > > We still don't have any > answer or comments about the VMware system VM (specifically this > template: http://download.cloud.com/templates/burbank/burbank- > systemvm-08012012.ova > >>> >> >
Re: [ACS41] System VMs not syncing time - does this block the release?
Chiradeep, As I think thought it, I don't think a documentation note will sufficient because the SSVM can be destroyed and respawned by CloudStack without intervention by a human. Therefore, we can get into a situation where an operator installs and configures NTP, and then at some point in the future, the SSVM gets respawned and clocks drift. The worst part about this scenario is that the failures for S3 sync become silent since we have no mechanism to surface the failure to a dashboard or monitoring system. How much effort (i.e. hours/days) would it be to build a new image? Thanks, -John On May 15, 2013, at 4:59 PM, Chiradeep Vittal wrote: > Did some further digging around as to why > /proc/sys/xen/independent_wallclock is not there on the Debian system vm. > > TLDR; the kernel is PvOps (I.e., just a regular kernel that works like a > PV kernel, not a specialized paravirt kernel). To eliminate > special-casing, the independent_wallclock feature was dropped. However, > this means that the domU clock is NO LONGER synch'ed to dom0 and NTP is > required on ANY domU. So the clock on the domU is only sync'ed at domU > creation time. I suspect Citrix XenServer might have some workaround > > > http://www.gossamer-threads.com/lists/xen/users/234750 > > > What this means is that we *may* need to run ntp on system vms on Xen. > This is of course a non-trivial change involving updates to systemvms. > > I suspect that it does not matter much for virtual routers or console > proxy vms. > > We could have an advisory that for those users that care (e.g., those > using S3 sync) that they need to run ntp after the SSVM has been created. > I.e, login to the SSVM and run > apt-get update > apt-get install ntp > > -- > Chiradeep > > > > On 5/15/13 1:11 PM, "Chiradeep Vittal" wrote: > >> For VMWare, the command >> vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a >> patch for /etc/init.d/cloud-early-config to enable it >> >> >> On 5/15/13 12:23 PM, "Chiradeep Vittal" >> wrote: >> >>> I am not sure why it is missing, but I will refer to >>> Citrix XenServer 6.0 Virtual Machine Installation Guide >>> >>> http://s.apache.org/YGn >>> >>> And quote >>> "Time Handling in Linux VMs >>> By default, the clocks in a Linux VM are synchronized to the clock >>> running >>> on the control domain, and cannot be >>> independently changed. This mode is a convenient default, since only the >>> control domain needs to be running the >>> NTP service to keep accurate time across all VMs. Upon installation of a >>> new Linux VM, make sure you change the >>> time-zone from the default UTC to your local value (see the section >>> called >>> ³Release Notes² for specific distribution >>> instructions). >>> To set individual Linux VMs to maintain independent times >>> 1. From a root prompt on the VM, run the command: echo 1 > >>> /proc/sys/xen/independent_wallclock >>> 2. This can be persisted across reboots by changing the /etc/sysctl.conf >>> configuration file and adding: >>> # Set independent wall clock time >>> xen.independent_wallclock=1 >>> 3. As a third alternative, independent_wallclock=1 may also be passed as >>> a >>> boot parameter to the VM. >>> " >>> >>> >>> On 5/15/13 12:09 PM, "Chip Childers" wrote: >>> On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: > /proc/sys is not a regular filesystem and cannot be added to from the > shell. > Drivers need to add nodes into this filesystem. Backing up a bit... This is the current system VM image that the 4.1 software should be using on Xen: http://download.cloud.com/templates/acton/acton- systemvm-02062012.vhd.bz2 Does this system VM have the correct drives installed to set /proc/sys/xen to use the Dom0 time for sync? If so, then is this a devcloud only issue for Xen? If that's the case, then we shouldn't block a release to simply improve things. We do know that a patch for kvm was needed. We still don't have any answer or comments about the VMware system VM (specifically this template: http://download.cloud.com/templates/burbank/burbank- systemvm-08012012.ova >>> >> >
Re: [ACS41] System VMs not syncing time - does this block the release?
On Wed, May 15, 2013 at 01:59:13PM -0700, Chiradeep Vittal wrote: > Did some further digging around as to why > /proc/sys/xen/independent_wallclock is not there on the Debian system vm. > > TLDR; the kernel is PvOps (I.e., just a regular kernel that works like a > PV kernel, not a specialized paravirt kernel). To eliminate > special-casing, the independent_wallclock feature was dropped. However, > this means that the domU clock is NO LONGER synch'ed to dom0 and NTP is > required on ANY domU. So the clock on the domU is only sync'ed at domU > creation time. I suspect Citrix XenServer might have some workaround > > > http://www.gossamer-threads.com/lists/xen/users/234750 > > > What this means is that we *may* need to run ntp on system vms on Xen. > This is of course a non-trivial change involving updates to systemvms. > > I suspect that it does not matter much for virtual routers or console > proxy vms. > > We could have an advisory that for those users that care (e.g., those > using S3 sync) that they need to run ntp after the SSVM has been created. > I.e, login to the SSVM and run > apt-get update > apt-get install ntp Fair idea, but what about when the MS decides that it needs more SSVMs? Is another possibility to take the existing system VM templates and simply update them for this issue? The potential benefit of either approach above is that we would be able to ensure that we did it on all three templates if required. We also obviously need to consider what to do for 4.2, where we were planning on officially releasing new system VMs. > > -- > Chiradeep > > > > On 5/15/13 1:11 PM, "Chiradeep Vittal" wrote: > > >For VMWare, the command > >vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a > >patch for /etc/init.d/cloud-early-config to enable it > > > > > >On 5/15/13 12:23 PM, "Chiradeep Vittal" > >wrote: > > > >>I am not sure why it is missing, but I will refer to > >>Citrix XenServer 6.0 Virtual Machine Installation Guide > >> > >>http://s.apache.org/YGn > >> > >>And quote > >>"Time Handling in Linux VMs > >>By default, the clocks in a Linux VM are synchronized to the clock > >>running > >>on the control domain, and cannot be > >>independently changed. This mode is a convenient default, since only the > >>control domain needs to be running the > >>NTP service to keep accurate time across all VMs. Upon installation of a > >>new Linux VM, make sure you change the > >>time-zone from the default UTC to your local value (see the section > >>called > >>³Release Notes² for specific distribution > >>instructions). > >>To set individual Linux VMs to maintain independent times > >>1. From a root prompt on the VM, run the command: echo 1 > > >>/proc/sys/xen/independent_wallclock > >>2. This can be persisted across reboots by changing the /etc/sysctl.conf > >>configuration file and adding: > >># Set independent wall clock time > >>xen.independent_wallclock=1 > >>3. As a third alternative, independent_wallclock=1 may also be passed as > >>a > >>boot parameter to the VM. > >>" > >> > >> > >>On 5/15/13 12:09 PM, "Chip Childers" wrote: > >> > >>>On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: > /proc/sys is not a regular filesystem and cannot be added to from the > shell. > Drivers need to add nodes into this filesystem. > >>> > >>>Backing up a bit... > >>> > >>>This is the current system VM image that the 4.1 software should be > >>>using on Xen: > >>> > >>>http://download.cloud.com/templates/acton/acton- > >>>systemvm-02062012.vhd.bz2 > >>> > >>>Does this system VM have the correct drives installed to set > >>>/proc/sys/xen to use the Dom0 time for sync? > >>> > >>>If so, then is this a devcloud only issue for Xen? If that's the case, > >>>then we shouldn't block a release to simply improve things. > >>> > >>>We do know that a patch for kvm was needed. > >>> > >>>We still don't have any > >>>answer or comments about the VMware system VM (specifically this > >>>template: http://download.cloud.com/templates/burbank/burbank- > >>>systemvm-08012012.ova > >>> > >> > > >
Re: [ACS41] System VMs not syncing time - does this block the release?
Did some further digging around as to why /proc/sys/xen/independent_wallclock is not there on the Debian system vm. TLDR; the kernel is PvOps (I.e., just a regular kernel that works like a PV kernel, not a specialized paravirt kernel). To eliminate special-casing, the independent_wallclock feature was dropped. However, this means that the domU clock is NO LONGER synch'ed to dom0 and NTP is required on ANY domU. So the clock on the domU is only sync'ed at domU creation time. I suspect Citrix XenServer might have some workaround http://www.gossamer-threads.com/lists/xen/users/234750 What this means is that we *may* need to run ntp on system vms on Xen. This is of course a non-trivial change involving updates to systemvms. I suspect that it does not matter much for virtual routers or console proxy vms. We could have an advisory that for those users that care (e.g., those using S3 sync) that they need to run ntp after the SSVM has been created. I.e, login to the SSVM and run apt-get update apt-get install ntp -- Chiradeep On 5/15/13 1:11 PM, "Chiradeep Vittal" wrote: >For VMWare, the command >vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a >patch for /etc/init.d/cloud-early-config to enable it > > >On 5/15/13 12:23 PM, "Chiradeep Vittal" >wrote: > >>I am not sure why it is missing, but I will refer to >>Citrix XenServer 6.0 Virtual Machine Installation Guide >> >>http://s.apache.org/YGn >> >>And quote >>"Time Handling in Linux VMs >>By default, the clocks in a Linux VM are synchronized to the clock >>running >>on the control domain, and cannot be >>independently changed. This mode is a convenient default, since only the >>control domain needs to be running the >>NTP service to keep accurate time across all VMs. Upon installation of a >>new Linux VM, make sure you change the >>time-zone from the default UTC to your local value (see the section >>called >>³Release Notes² for specific distribution >>instructions). >>To set individual Linux VMs to maintain independent times >>1. From a root prompt on the VM, run the command: echo 1 > >>/proc/sys/xen/independent_wallclock >>2. This can be persisted across reboots by changing the /etc/sysctl.conf >>configuration file and adding: >># Set independent wall clock time >>xen.independent_wallclock=1 >>3. As a third alternative, independent_wallclock=1 may also be passed as >>a >>boot parameter to the VM. >>" >> >> >>On 5/15/13 12:09 PM, "Chip Childers" wrote: >> >>>On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: /proc/sys is not a regular filesystem and cannot be added to from the shell. Drivers need to add nodes into this filesystem. >>> >>>Backing up a bit... >>> >>>This is the current system VM image that the 4.1 software should be >>>using on Xen: >>> >>>http://download.cloud.com/templates/acton/acton- >>>systemvm-02062012.vhd.bz2 >>> >>>Does this system VM have the correct drives installed to set >>>/proc/sys/xen to use the Dom0 time for sync? >>> >>>If so, then is this a devcloud only issue for Xen? If that's the case, >>>then we shouldn't block a release to simply improve things. >>> >>>We do know that a patch for kvm was needed. >>> >>>We still don't have any >>>answer or comments about the VMware system VM (specifically this >>>template: http://download.cloud.com/templates/burbank/burbank- >>>systemvm-08012012.ova >>> >> >
Re: [ACS41] System VMs not syncing time - does this block the release?
For VMWare, the command vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a patch for /etc/init.d/cloud-early-config to enable it On 5/15/13 12:23 PM, "Chiradeep Vittal" wrote: >I am not sure why it is missing, but I will refer to >Citrix XenServer 6.0 Virtual Machine Installation Guide > >http://s.apache.org/YGn > >And quote >"Time Handling in Linux VMs >By default, the clocks in a Linux VM are synchronized to the clock running >on the control domain, and cannot be >independently changed. This mode is a convenient default, since only the >control domain needs to be running the >NTP service to keep accurate time across all VMs. Upon installation of a >new Linux VM, make sure you change the >time-zone from the default UTC to your local value (see the section called >³Release Notes² for specific distribution >instructions). >To set individual Linux VMs to maintain independent times >1. From a root prompt on the VM, run the command: echo 1 > >/proc/sys/xen/independent_wallclock >2. This can be persisted across reboots by changing the /etc/sysctl.conf >configuration file and adding: ># Set independent wall clock time >xen.independent_wallclock=1 >3. As a third alternative, independent_wallclock=1 may also be passed as a >boot parameter to the VM. >" > > >On 5/15/13 12:09 PM, "Chip Childers" wrote: > >>On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: >>> /proc/sys is not a regular filesystem and cannot be added to from the >>> shell. >>> Drivers need to add nodes into this filesystem. >> >>Backing up a bit... >> >>This is the current system VM image that the 4.1 software should be >>using on Xen: >> >>http://download.cloud.com/templates/acton/acton- >>systemvm-02062012.vhd.bz2 >> >>Does this system VM have the correct drives installed to set >>/proc/sys/xen to use the Dom0 time for sync? >> >>If so, then is this a devcloud only issue for Xen? If that's the case, >>then we shouldn't block a release to simply improve things. >> >>We do know that a patch for kvm was needed. >> >>We still don't have any >>answer or comments about the VMware system VM (specifically this >>template: http://download.cloud.com/templates/burbank/burbank- >>systemvm-08012012.ova >> >
Re: [ACS41] System VMs not syncing time - does this block the release?
I am not sure why it is missing, but I will refer to Citrix XenServer 6.0 Virtual Machine Installation Guide http://s.apache.org/YGn And quote "Time Handling in Linux VMs By default, the clocks in a Linux VM are synchronized to the clock running on the control domain, and cannot be independently changed. This mode is a convenient default, since only the control domain needs to be running the NTP service to keep accurate time across all VMs. Upon installation of a new Linux VM, make sure you change the time-zone from the default UTC to your local value (see the section called ³Release Notes² for specific distribution instructions). To set individual Linux VMs to maintain independent times 1. From a root prompt on the VM, run the command: echo 1 > /proc/sys/xen/independent_wallclock 2. This can be persisted across reboots by changing the /etc/sysctl.conf configuration file and adding: # Set independent wall clock time xen.independent_wallclock=1 3. As a third alternative, independent_wallclock=1 may also be passed as a boot parameter to the VM. " On 5/15/13 12:09 PM, "Chip Childers" wrote: >On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: >> /proc/sys is not a regular filesystem and cannot be added to from the >> shell. >> Drivers need to add nodes into this filesystem. > >Backing up a bit... > >This is the current system VM image that the 4.1 software should be >using on Xen: > >http://download.cloud.com/templates/acton/acton- >systemvm-02062012.vhd.bz2 > >Does this system VM have the correct drives installed to set >/proc/sys/xen to use the Dom0 time for sync? > >If so, then is this a devcloud only issue for Xen? If that's the case, >then we shouldn't block a release to simply improve things. > >We do know that a patch for kvm was needed. > >We still don't have any >answer or comments about the VMware system VM (specifically this >template: http://download.cloud.com/templates/burbank/burbank- >systemvm-08012012.ova >
Re: [ACS41] System VMs not syncing time - does this block the release?
On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: > /proc/sys is not a regular filesystem and cannot be added to from the > shell. > Drivers need to add nodes into this filesystem. Backing up a bit... This is the current system VM image that the 4.1 software should be using on Xen: http://download.cloud.com/templates/acton/acton- systemvm-02062012.vhd.bz2 Does this system VM have the correct drives installed to set /proc/sys/xen to use the Dom0 time for sync? If so, then is this a devcloud only issue for Xen? If that's the case, then we shouldn't block a release to simply improve things. We do know that a patch for kvm was needed. We still don't have any answer or comments about the VMware system VM (specifically this template: http://download.cloud.com/templates/burbank/burbank- systemvm-08012012.ova
Re: [ACS41] System VMs not syncing time - does this block the release?
/proc/sys is not a regular filesystem and cannot be added to from the shell. Drivers need to add nodes into this filesystem. On 5/15/13 11:50 AM, "Chip Childers" wrote: >On Wed, May 15, 2013 at 02:48:13PM -0400, John Burwell wrote: >> All of these things being said, it appears that the Xen behavior may be >>a regression that can be addressed with a relatively straightforward fix >>(dropping the proper file in /proc/sys/xen), > >Agreed. Anyone want to submit the patch for this? > >> and KVM has been fixed by Marcus. > >Yes, and applied to master and 4.1 > >> Is anyone looking at a fix for VMWare? > >Not that I'm aware of right now. Can anyone take this? > >-chip
Re: [ACS41] System VMs not syncing time - does this block the release?
On Wed, May 15, 2013 at 02:48:13PM -0400, John Burwell wrote: > All of these things being said, it appears that the Xen behavior may be a > regression that can be addressed with a relatively straightforward fix > (dropping the proper file in /proc/sys/xen), Agreed. Anyone want to submit the patch for this? > and KVM has been fixed by Marcus. Yes, and applied to master and 4.1 > Is anyone looking at a fix for VMWare? Not that I'm aware of right now. Can anyone take this? -chip
Re: [ACS41] System VMs not syncing time - does this block the release?
Chiradeep, As I mentioned earlier, this issue is larger than S3-backed Secondary Storage. It just happens that this issue was surfaced by testing that feature.Clock drift exceeding than a few seconds can be operational issue (e.g. file timestamps, logging, etc). A lack of reliable clock sync will be an issue in any accredited environment (e.g. SOX) due to its impact on the integrity on audit trails. Clock drift in virtualized environments can easily exceed 15 minutes in either direction. The drift behavior is driven by a number of environmental variables. In my experience, I have seen clocks drift by hours in less than 30 minutes of real time. Therefore, I would caution against taking a single environment as a general benchmark. VMWare has a written white paper on the subject [1] which goes into a great deal of depth around system clocks in virtualized environments. All of these things being said, it appears that the Xen behavior may be a regression that can be addressed with a relatively straightforward fix (dropping the proper file in /proc/sys/xen), and KVM has been fixed by Marcus. Is anyone looking at a fix for VMWare? Thanks, -John [1]: http://www.vmware.com/files/pdf/techpaper/Timekeeping-In-VirtualMachines.pdf On May 15, 2013, at 2:29 PM, Chiradeep Vittal wrote: > The previous ones were on XS 5.6 FP2 > This one's on XS 6.0.2 > r-275166-VM 18:22:10 up 8 days, > domU: Wed May 15 18:22:10 UTC 2013 > dom0: Wed May 15 11:22:10 PDT 2013 > > > > On 5/15/13 11:22 AM, "Chiradeep Vittal" > wrote: > >> The normal S3 time sync is 15 minutes. I can't imagine a drift of 15 >> minutes in a few days of operation? I logged into 3 system vms running on >> Xen and saw this drift: >> >> r-9-VM 17:52:37 up 29 days, 10:33, >> domU: Wed May 15 17:52:37 UTC 2013 >> dom0: Wed May 15 10:52:37 PDT 2013 >> >> r-535-VM 18:13:46 up 43 days, 20:49, >> domU: Wed May 15 18:13:46 UTC 2013 >> dom0: Wed May 15 11:14:47 PDT 2013 >> >> r-247793-VM 18:18:20 up 43 days, 20:53, >> domU: Wed May 15 18:18:20 UTC 2013 >> dom0: Wed May 15 11:18:33 PDT 2013 >> >> >> A PV kernel such as the systemvm's automatically maintains the clock sync >> with dom0. >> >> >> On 5/15/13 10:30 AM, "John Burwell" wrote: >> >>> Chiradeep, >>> >>> The issue I am experiencing is that the system VMs are not syncing to >>> dom0 >>> on devcloud (i.e. the dom0 clock and the SSVM clock are different). As I >>> mentioned earlier in this thread, the syncing was working previously >>> which >>> seems to jibe with your findings. What mechanism is used to sync the >>> dom0 >>> and domU clocks (e.g. NTP, kernel driver, etc)? It may be a situation >>> where the pieces are present, but they aren't configured properly or >>> simply >>> not running. >>> >>> As an aside, we can not run VirtualBox Additions on devcloud due a >>> conflict >>> with the Xen kernel. Therefore, I hard execute "ntpdate pool.ntp.org" >>> periodically on the devcloud host to keep the clock synced with the "real >>> world". Another approach is to configure NTP with a very large drift and >>> increase check frequency to accomodate the large clock swings that can >>> occur. >>> >>> Thanks, >>> -John >>> >>> >>> On Wed, May 15, 2013 at 1:21 PM, Chiradeep Vittal < >>> chiradeep.vit...@citrix.com> wrote: >>> Perhaps this is a problem with DevCloud? http://nerdboys.com/2011/03/15/how-to-fix-virtualbox-time-synchonization - pr oblems/ On 5/15/13 10:17 AM, "Chiradeep Vittal" wrote: > According to our resident Xen expert, any PV kernel automatically syncs to > the hardware clock on dom0. > > On 5/15/13 9:50 AM, "John Burwell" wrote: > >> Marcus, >> >> Agreed. I think we need to add a set of hypervisor agnostic time >> keeping guidelines to the documentation. I just wanted to make sure >> there wasn't anything KVM specific that should be added as well. >> >> Thanks, >> -John >> >> On May 15, 2013, at 12:48 PM, Marcus Sorensen >> wrote: >> >>> Just the general one that system vms sync their time to the >>> hypervisor, thus admins need to keep the hypervisor time correct. It >>> sounds like that will be the case for all three, if we can manage it. >>> >>> On Wed, May 15, 2013 at 10:44 AM, John Burwell >>> wrote: Marcus, Excellent. So, it looks like we have KVM resolved. We just need to address Xen and VMWare now. Do you think we need to any guidance to the documentation regarding KVM time keeping (e.g. environmental prerequisites)? Thanks, -John On May 15, 2013, at 12:39 PM, Marcus Sorensen wrote: > KVM LibvirtComputingResource has been patched in master. Tested on > master, 4.1, and both the ac
Re: [ACS41] System VMs not syncing time - does this block the release?
The previous ones were on XS 5.6 FP2 This one's on XS 6.0.2 r-275166-VM 18:22:10 up 8 days, domU: Wed May 15 18:22:10 UTC 2013 dom0: Wed May 15 11:22:10 PDT 2013 On 5/15/13 11:22 AM, "Chiradeep Vittal" wrote: >The normal S3 time sync is 15 minutes. I can't imagine a drift of 15 >minutes in a few days of operation? I logged into 3 system vms running on >Xen and saw this drift: > >r-9-VM 17:52:37 up 29 days, 10:33, >domU: Wed May 15 17:52:37 UTC 2013 >dom0: Wed May 15 10:52:37 PDT 2013 > >r-535-VM 18:13:46 up 43 days, 20:49, >domU: Wed May 15 18:13:46 UTC 2013 >dom0: Wed May 15 11:14:47 PDT 2013 > >r-247793-VM 18:18:20 up 43 days, 20:53, >domU: Wed May 15 18:18:20 UTC 2013 >dom0: Wed May 15 11:18:33 PDT 2013 > > >A PV kernel such as the systemvm's automatically maintains the clock sync >with dom0. > > >On 5/15/13 10:30 AM, "John Burwell" wrote: > >>Chiradeep, >> >>The issue I am experiencing is that the system VMs are not syncing to >>dom0 >>on devcloud (i.e. the dom0 clock and the SSVM clock are different). As I >>mentioned earlier in this thread, the syncing was working previously >>which >>seems to jibe with your findings. What mechanism is used to sync the >>dom0 >>and domU clocks (e.g. NTP, kernel driver, etc)? It may be a situation >>where the pieces are present, but they aren't configured properly or >>simply >>not running. >> >>As an aside, we can not run VirtualBox Additions on devcloud due a >>conflict >>with the Xen kernel. Therefore, I hard execute "ntpdate pool.ntp.org" >>periodically on the devcloud host to keep the clock synced with the "real >>world". Another approach is to configure NTP with a very large drift and >>increase check frequency to accomodate the large clock swings that can >>occur. >> >>Thanks, >>-John >> >> >>On Wed, May 15, 2013 at 1:21 PM, Chiradeep Vittal < >>chiradeep.vit...@citrix.com> wrote: >> >>> Perhaps this is a problem with DevCloud? >>> >>>http://nerdboys.com/2011/03/15/how-to-fix-virtualbox-time-synchonization >>>- >>>pr >>> oblems/ >>> >>> >>> >>> On 5/15/13 10:17 AM, "Chiradeep Vittal" >>> wrote: >>> >>> >According to our resident Xen expert, any PV kernel automatically >>>syncs to >>> >the hardware clock on dom0. >>> > >>> >On 5/15/13 9:50 AM, "John Burwell" wrote: >>> > >>> >>Marcus, >>> >> >>> >>Agreed. I think we need to add a set of hypervisor agnostic time >>> >>keeping guidelines to the documentation. I just wanted to make sure >>> >>there wasn't anything KVM specific that should be added as well. >>> >> >>> >>Thanks, >>> >>-John >>> >> >>> >>On May 15, 2013, at 12:48 PM, Marcus Sorensen >>> >>wrote: >>> >> >>> >>> Just the general one that system vms sync their time to the >>> >>> hypervisor, thus admins need to keep the hypervisor time correct. >>>It >>> >>> sounds like that will be the case for all three, if we can manage >>>it. >>> >>> >>> >>> On Wed, May 15, 2013 at 10:44 AM, John Burwell >>> >>>wrote: >>> Marcus, >>> >>> Excellent. So, it looks like we have KVM resolved. We just need >>>to >>> address Xen and VMWare now. Do you think we need to any guidance >>>to >>> the documentation regarding KVM time keeping (e.g. environmental >>> prerequisites)? >>> >>> Thanks, >>> -John >>> >>> >>> >>> On May 15, 2013, at 12:39 PM, Marcus Sorensen >>> >>> wrote: >>> >>> > KVM LibvirtComputingResource has been patched in master. Tested >>>on >>> > master, 4.1, and both the acton and current system vm templates. >>>This >>> > patch makes system vms use 'kvmclock' for their timer, which is a >>>vm >>> > driver that gets it's time from the hypervisor. No change to the >>> > system vm template itself. >>> > >>> > bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 >>> > >>> > On Wed, May 15, 2013 at 9:08 AM, Chip Childers >>> > wrote: >>> >> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: >>> >>> Chip, >>> >>> >>> >>> One other item I neglected to mention was that clock sync, at >>>least >>> >>>for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. >>> >>>Previously when I encountered these issues, syncing the host's >>>clock >>> >>>and rebuilding the system VMs addressed the issue. I assumed, >>>but >>> >>>never verified, that the SSVM was syncing back against the >>>host's >>> >>>clock through hypervisor. During my testing yesterday, aside >>>from >>> >>>hard setting the clock, I was unable to force clock sync on the >>> >>>SSVM. >>> >>> >>> >>> Thanks, >>> >>> -John >>> >> >>> >> I think that's our issue right now... answering the question: >>>Why >>> >>is >>> >> this only an issue now? Did we just get lucky up to this point? >>> >>Since >>> >> the SSVMs are the same template as the timeframe you mention, I >>>tend >>> >>to >>> >> believe that you / we were just lucky. >>> >> >>> >> Anyone else have thoughts? >>> >> >>>
Re: [ACS41] System VMs not syncing time - does this block the release?
The normal S3 time sync is 15 minutes. I can't imagine a drift of 15 minutes in a few days of operation? I logged into 3 system vms running on Xen and saw this drift: r-9-VM 17:52:37 up 29 days, 10:33, domU: Wed May 15 17:52:37 UTC 2013 dom0: Wed May 15 10:52:37 PDT 2013 r-535-VM 18:13:46 up 43 days, 20:49, domU: Wed May 15 18:13:46 UTC 2013 dom0: Wed May 15 11:14:47 PDT 2013 r-247793-VM 18:18:20 up 43 days, 20:53, domU: Wed May 15 18:18:20 UTC 2013 dom0: Wed May 15 11:18:33 PDT 2013 A PV kernel such as the systemvm's automatically maintains the clock sync with dom0. On 5/15/13 10:30 AM, "John Burwell" wrote: >Chiradeep, > >The issue I am experiencing is that the system VMs are not syncing to dom0 >on devcloud (i.e. the dom0 clock and the SSVM clock are different). As I >mentioned earlier in this thread, the syncing was working previously which >seems to jibe with your findings. What mechanism is used to sync the dom0 >and domU clocks (e.g. NTP, kernel driver, etc)? It may be a situation >where the pieces are present, but they aren't configured properly or >simply >not running. > >As an aside, we can not run VirtualBox Additions on devcloud due a >conflict >with the Xen kernel. Therefore, I hard execute "ntpdate pool.ntp.org" >periodically on the devcloud host to keep the clock synced with the "real >world". Another approach is to configure NTP with a very large drift and >increase check frequency to accomodate the large clock swings that can >occur. > >Thanks, >-John > > >On Wed, May 15, 2013 at 1:21 PM, Chiradeep Vittal < >chiradeep.vit...@citrix.com> wrote: > >> Perhaps this is a problem with DevCloud? >> >>http://nerdboys.com/2011/03/15/how-to-fix-virtualbox-time-synchonization- >>pr >> oblems/ >> >> >> >> On 5/15/13 10:17 AM, "Chiradeep Vittal" >> wrote: >> >> >According to our resident Xen expert, any PV kernel automatically >>syncs to >> >the hardware clock on dom0. >> > >> >On 5/15/13 9:50 AM, "John Burwell" wrote: >> > >> >>Marcus, >> >> >> >>Agreed. I think we need to add a set of hypervisor agnostic time >> >>keeping guidelines to the documentation. I just wanted to make sure >> >>there wasn't anything KVM specific that should be added as well. >> >> >> >>Thanks, >> >>-John >> >> >> >>On May 15, 2013, at 12:48 PM, Marcus Sorensen >> >>wrote: >> >> >> >>> Just the general one that system vms sync their time to the >> >>> hypervisor, thus admins need to keep the hypervisor time correct. It >> >>> sounds like that will be the case for all three, if we can manage >>it. >> >>> >> >>> On Wed, May 15, 2013 at 10:44 AM, John Burwell >> >>>wrote: >> Marcus, >> >> Excellent. So, it looks like we have KVM resolved. We just need >>to >> address Xen and VMWare now. Do you think we need to any guidance to >> the documentation regarding KVM time keeping (e.g. environmental >> prerequisites)? >> >> Thanks, >> -John >> >> >> >> On May 15, 2013, at 12:39 PM, Marcus Sorensen >> wrote: >> >> > KVM LibvirtComputingResource has been patched in master. Tested on >> > master, 4.1, and both the acton and current system vm templates. >>This >> > patch makes system vms use 'kvmclock' for their timer, which is a >>vm >> > driver that gets it's time from the hypervisor. No change to the >> > system vm template itself. >> > >> > bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 >> > >> > On Wed, May 15, 2013 at 9:08 AM, Chip Childers >> > wrote: >> >> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: >> >>> Chip, >> >>> >> >>> One other item I neglected to mention was that clock sync, at >>least >> >>>for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. >> >>>Previously when I encountered these issues, syncing the host's >>clock >> >>>and rebuilding the system VMs addressed the issue. I assumed, >>but >> >>>never verified, that the SSVM was syncing back against the host's >> >>>clock through hypervisor. During my testing yesterday, aside >>from >> >>>hard setting the clock, I was unable to force clock sync on the >> >>>SSVM. >> >>> >> >>> Thanks, >> >>> -John >> >> >> >> I think that's our issue right now... answering the question: >>Why >> >>is >> >> this only an issue now? Did we just get lucky up to this point? >> >>Since >> >> the SSVMs are the same template as the timeframe you mention, I >>tend >> >>to >> >> believe that you / we were just lucky. >> >> >> >> Anyone else have thoughts? >> >> >> >>> >> >>> On May 15, 2013, at 10:18 AM, Chip Childers >> >>> wrote: >> >>> >> Starting a thread on this specific issue. >> >> CLOUDSTACK-2492 was opened, which is basically the fact that >>the >> System >> VMs aren't syncing time to the host or to an NTP server. The >>S3 >> integration is broken because of this
Re: [ACS41] System VMs not syncing time - does this block the release?
Anthony, I checked the 4.1 system VM template (downloaded per the latest installation instructions from Jenkins), and it also missing this file. Thanks, -John On May 15, 2013, at 2:03 PM, Anthony Xu wrote: > In 3.0.x system VM template, /proc/sys/xen/independent_wallclock is set to 0, > that means system VM is using dom0 time, it is always synced with dom0 time. > In 4.2 system VM template, /proc/sys/xen/independent_wallclock doesn't > exist, looks like it is 1 by default by checking > http://wiki.xensource.com/wiki/Xen_FAQ_DomU#How_can_i_synchronize_a_dom0_clock.3F, > which might cause time drift, maybe we need to enable NTP inside system VM. > > Anthony > > > > > -Original Message- > From: John Burwell [mailto:jburw...@basho.com] > Sent: Wednesday, May 15, 2013 10:30 AM > To: dev@cloudstack.apache.org > Subject: Re: [ACS41] System VMs not syncing time - does this block the > release? > > Chiradeep, > > The issue I am experiencing is that the system VMs are not syncing to dom0 on > devcloud (i.e. the dom0 clock and the SSVM clock are different). As I > mentioned earlier in this thread, the syncing was working previously which > seems to jibe with your findings. What mechanism is used to sync the dom0 > and domU clocks (e.g. NTP, kernel driver, etc)? It may be a situation where > the pieces are present, but they aren't configured properly or simply not > running. > > As an aside, we can not run VirtualBox Additions on devcloud due a conflict > with the Xen kernel. Therefore, I hard execute "ntpdate pool.ntp.org" > periodically on the devcloud host to keep the clock synced with the "real > world". Another approach is to configure NTP with a very large drift and > increase check frequency to accomodate the large clock swings that can occur. > > Thanks, > -John > > > On Wed, May 15, 2013 at 1:21 PM, Chiradeep Vittal < > chiradeep.vit...@citrix.com> wrote: > >> Perhaps this is a problem with DevCloud? >> http://nerdboys.com/2011/03/15/how-to-fix-virtualbox-time-synchonizati >> on-pr >> oblems/ >> >> >> >> On 5/15/13 10:17 AM, "Chiradeep Vittal" >> wrote: >> >>> According to our resident Xen expert, any PV kernel automatically >>> syncs to the hardware clock on dom0. >>> >>> On 5/15/13 9:50 AM, "John Burwell" wrote: >>> >>>> Marcus, >>>> >>>> Agreed. I think we need to add a set of hypervisor agnostic time >>>> keeping guidelines to the documentation. I just wanted to make sure >>>> there wasn't anything KVM specific that should be added as well. >>>> >>>> Thanks, >>>> -John >>>> >>>> On May 15, 2013, at 12:48 PM, Marcus Sorensen >>>> wrote: >>>> >>>>> Just the general one that system vms sync their time to the >>>>> hypervisor, thus admins need to keep the hypervisor time correct. >>>>> It sounds like that will be the case for all three, if we can manage it. >>>>> >>>>> On Wed, May 15, 2013 at 10:44 AM, John Burwell >>>>> >>>>> wrote: >>>>>> Marcus, >>>>>> >>>>>> Excellent. So, it looks like we have KVM resolved. We just need >>>>>> to address Xen and VMWare now. Do you think we need to any >>>>>> guidance to the documentation regarding KVM time keeping (e.g. >>>>>> environmental prerequisites)? >>>>>> >>>>>> Thanks, >>>>>> -John >>>>>> >>>>>> >>>>>> >>>>>> On May 15, 2013, at 12:39 PM, Marcus Sorensen >>>>>> >>>>>> wrote: >>>>>> >>>>>>> KVM LibvirtComputingResource has been patched in master. Tested >>>>>>> on master, 4.1, and both the acton and current system vm >>>>>>> templates. This patch makes system vms use 'kvmclock' for their >>>>>>> timer, which is a vm driver that gets it's time from the >>>>>>> hypervisor. No change to the system vm template itself. >>>>>>> >>>>>>> bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 >>>>>>> >>>>>>> On Wed, May 15, 2013 at 9:08 AM, Chip Childers >>>>>>> wrote: >>>>>>>> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwe
RE: [ACS41] System VMs not syncing time - does this block the release?
In 3.0.x system VM template, /proc/sys/xen/independent_wallclock is set to 0, that means system VM is using dom0 time, it is always synced with dom0 time. In 4.2 system VM template, /proc/sys/xen/independent_wallclock doesn't exist, looks like it is 1 by default by checking http://wiki.xensource.com/wiki/Xen_FAQ_DomU#How_can_i_synchronize_a_dom0_clock.3F, which might cause time drift, maybe we need to enable NTP inside system VM. Anthony -Original Message- From: John Burwell [mailto:jburw...@basho.com] Sent: Wednesday, May 15, 2013 10:30 AM To: dev@cloudstack.apache.org Subject: Re: [ACS41] System VMs not syncing time - does this block the release? Chiradeep, The issue I am experiencing is that the system VMs are not syncing to dom0 on devcloud (i.e. the dom0 clock and the SSVM clock are different). As I mentioned earlier in this thread, the syncing was working previously which seems to jibe with your findings. What mechanism is used to sync the dom0 and domU clocks (e.g. NTP, kernel driver, etc)? It may be a situation where the pieces are present, but they aren't configured properly or simply not running. As an aside, we can not run VirtualBox Additions on devcloud due a conflict with the Xen kernel. Therefore, I hard execute "ntpdate pool.ntp.org" periodically on the devcloud host to keep the clock synced with the "real world". Another approach is to configure NTP with a very large drift and increase check frequency to accomodate the large clock swings that can occur. Thanks, -John On Wed, May 15, 2013 at 1:21 PM, Chiradeep Vittal < chiradeep.vit...@citrix.com> wrote: > Perhaps this is a problem with DevCloud? > http://nerdboys.com/2011/03/15/how-to-fix-virtualbox-time-synchonizati > on-pr > oblems/ > > > > On 5/15/13 10:17 AM, "Chiradeep Vittal" > wrote: > > >According to our resident Xen expert, any PV kernel automatically > >syncs to the hardware clock on dom0. > > > >On 5/15/13 9:50 AM, "John Burwell" wrote: > > > >>Marcus, > >> > >>Agreed. I think we need to add a set of hypervisor agnostic time > >>keeping guidelines to the documentation. I just wanted to make sure > >>there wasn't anything KVM specific that should be added as well. > >> > >>Thanks, > >>-John > >> > >>On May 15, 2013, at 12:48 PM, Marcus Sorensen > >>wrote: > >> > >>> Just the general one that system vms sync their time to the > >>> hypervisor, thus admins need to keep the hypervisor time correct. > >>> It sounds like that will be the case for all three, if we can manage it. > >>> > >>> On Wed, May 15, 2013 at 10:44 AM, John Burwell > >>> > >>>wrote: > >>>> Marcus, > >>>> > >>>> Excellent. So, it looks like we have KVM resolved. We just need > >>>>to address Xen and VMWare now. Do you think we need to any > >>>>guidance to the documentation regarding KVM time keeping (e.g. > >>>>environmental prerequisites)? > >>>> > >>>> Thanks, > >>>> -John > >>>> > >>>> > >>>> > >>>> On May 15, 2013, at 12:39 PM, Marcus Sorensen > >>>> > >>>>wrote: > >>>> > >>>>> KVM LibvirtComputingResource has been patched in master. Tested > >>>>> on master, 4.1, and both the acton and current system vm > >>>>> templates. This patch makes system vms use 'kvmclock' for their > >>>>> timer, which is a vm driver that gets it's time from the > >>>>> hypervisor. No change to the system vm template itself. > >>>>> > >>>>> bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 > >>>>> > >>>>> On Wed, May 15, 2013 at 9:08 AM, Chip Childers > >>>>> wrote: > >>>>>> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: > >>>>>>> Chip, > >>>>>>> > >>>>>>> One other item I neglected to mention was that clock sync, at > >>>>>>>least for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. > >>>>>>>Previously when I encountered these issues, syncing the host's > >>>>>>>clock and rebuilding the system VMs addressed the issue. I > >>>>>>>assumed, but never verified, that the SSVM was syncing back > >>>>>>>against the host's clock through hypervi
Re: [ACS41] System VMs not syncing time - does this block the release?
Chiradeep, The issue I am experiencing is that the system VMs are not syncing to dom0 on devcloud (i.e. the dom0 clock and the SSVM clock are different). As I mentioned earlier in this thread, the syncing was working previously which seems to jibe with your findings. What mechanism is used to sync the dom0 and domU clocks (e.g. NTP, kernel driver, etc)? It may be a situation where the pieces are present, but they aren't configured properly or simply not running. As an aside, we can not run VirtualBox Additions on devcloud due a conflict with the Xen kernel. Therefore, I hard execute "ntpdate pool.ntp.org" periodically on the devcloud host to keep the clock synced with the "real world". Another approach is to configure NTP with a very large drift and increase check frequency to accomodate the large clock swings that can occur. Thanks, -John On Wed, May 15, 2013 at 1:21 PM, Chiradeep Vittal < chiradeep.vit...@citrix.com> wrote: > Perhaps this is a problem with DevCloud? > http://nerdboys.com/2011/03/15/how-to-fix-virtualbox-time-synchonization-pr > oblems/ > > > > On 5/15/13 10:17 AM, "Chiradeep Vittal" > wrote: > > >According to our resident Xen expert, any PV kernel automatically syncs to > >the hardware clock on dom0. > > > >On 5/15/13 9:50 AM, "John Burwell" wrote: > > > >>Marcus, > >> > >>Agreed. I think we need to add a set of hypervisor agnostic time > >>keeping guidelines to the documentation. I just wanted to make sure > >>there wasn't anything KVM specific that should be added as well. > >> > >>Thanks, > >>-John > >> > >>On May 15, 2013, at 12:48 PM, Marcus Sorensen > >>wrote: > >> > >>> Just the general one that system vms sync their time to the > >>> hypervisor, thus admins need to keep the hypervisor time correct. It > >>> sounds like that will be the case for all three, if we can manage it. > >>> > >>> On Wed, May 15, 2013 at 10:44 AM, John Burwell > >>>wrote: > Marcus, > > Excellent. So, it looks like we have KVM resolved. We just need to > address Xen and VMWare now. Do you think we need to any guidance to > the documentation regarding KVM time keeping (e.g. environmental > prerequisites)? > > Thanks, > -John > > > > On May 15, 2013, at 12:39 PM, Marcus Sorensen > wrote: > > > KVM LibvirtComputingResource has been patched in master. Tested on > > master, 4.1, and both the acton and current system vm templates. This > > patch makes system vms use 'kvmclock' for their timer, which is a vm > > driver that gets it's time from the hypervisor. No change to the > > system vm template itself. > > > > bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 > > > > On Wed, May 15, 2013 at 9:08 AM, Chip Childers > > wrote: > >> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: > >>> Chip, > >>> > >>> One other item I neglected to mention was that clock sync, at least > >>>for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. > >>>Previously when I encountered these issues, syncing the host's clock > >>>and rebuilding the system VMs addressed the issue. I assumed, but > >>>never verified, that the SSVM was syncing back against the host's > >>>clock through hypervisor. During my testing yesterday, aside from > >>>hard setting the clock, I was unable to force clock sync on the > >>>SSVM. > >>> > >>> Thanks, > >>> -John > >> > >> I think that's our issue right now... answering the question: Why > >>is > >> this only an issue now? Did we just get lucky up to this point? > >>Since > >> the SSVMs are the same template as the timeframe you mention, I tend > >>to > >> believe that you / we were just lucky. > >> > >> Anyone else have thoughts? > >> > >>> > >>> On May 15, 2013, at 10:18 AM, Chip Childers > >>> wrote: > >>> > Starting a thread on this specific issue. > > CLOUDSTACK-2492 was opened, which is basically the fact that the > System > VMs aren't syncing time to the host or to an NTP server. The S3 > integration is broken because of this problem, and therefore could > not > be considered a function available in 4.1 if we release as is. > > We need input from people that know about the current system VMs > (the > 3.x VMs), as well as the possibility of using the newer ones that > we > have been considering experimental for 4.1.0. > > What should we do? > > -chip > >>> > >>> > > >> > > > >
Re: [ACS41] System VMs not syncing time - does this block the release?
Perhaps this is a problem with DevCloud? http://nerdboys.com/2011/03/15/how-to-fix-virtualbox-time-synchonization-pr oblems/ On 5/15/13 10:17 AM, "Chiradeep Vittal" wrote: >According to our resident Xen expert, any PV kernel automatically syncs to >the hardware clock on dom0. > >On 5/15/13 9:50 AM, "John Burwell" wrote: > >>Marcus, >> >>Agreed. I think we need to add a set of hypervisor agnostic time >>keeping guidelines to the documentation. I just wanted to make sure >>there wasn't anything KVM specific that should be added as well. >> >>Thanks, >>-John >> >>On May 15, 2013, at 12:48 PM, Marcus Sorensen >>wrote: >> >>> Just the general one that system vms sync their time to the >>> hypervisor, thus admins need to keep the hypervisor time correct. It >>> sounds like that will be the case for all three, if we can manage it. >>> >>> On Wed, May 15, 2013 at 10:44 AM, John Burwell >>>wrote: Marcus, Excellent. So, it looks like we have KVM resolved. We just need to address Xen and VMWare now. Do you think we need to any guidance to the documentation regarding KVM time keeping (e.g. environmental prerequisites)? Thanks, -John On May 15, 2013, at 12:39 PM, Marcus Sorensen wrote: > KVM LibvirtComputingResource has been patched in master. Tested on > master, 4.1, and both the acton and current system vm templates. This > patch makes system vms use 'kvmclock' for their timer, which is a vm > driver that gets it's time from the hypervisor. No change to the > system vm template itself. > > bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 > > On Wed, May 15, 2013 at 9:08 AM, Chip Childers > wrote: >> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: >>> Chip, >>> >>> One other item I neglected to mention was that clock sync, at least >>>for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. >>>Previously when I encountered these issues, syncing the host's clock >>>and rebuilding the system VMs addressed the issue. I assumed, but >>>never verified, that the SSVM was syncing back against the host's >>>clock through hypervisor. During my testing yesterday, aside from >>>hard setting the clock, I was unable to force clock sync on the >>>SSVM. >>> >>> Thanks, >>> -John >> >> I think that's our issue right now... answering the question: Why >>is >> this only an issue now? Did we just get lucky up to this point? >>Since >> the SSVMs are the same template as the timeframe you mention, I tend >>to >> believe that you / we were just lucky. >> >> Anyone else have thoughts? >> >>> >>> On May 15, 2013, at 10:18 AM, Chip Childers >>> wrote: >>> Starting a thread on this specific issue. CLOUDSTACK-2492 was opened, which is basically the fact that the System VMs aren't syncing time to the host or to an NTP server. The S3 integration is broken because of this problem, and therefore could not be considered a function available in 4.1 if we release as is. We need input from people that know about the current system VMs (the 3.x VMs), as well as the possibility of using the newer ones that we have been considering experimental for 4.1.0. What should we do? -chip >>> >>> >> >
Re: [ACS41] System VMs not syncing time - does this block the release?
According to our resident Xen expert, any PV kernel automatically syncs to the hardware clock on dom0. On 5/15/13 9:50 AM, "John Burwell" wrote: >Marcus, > >Agreed. I think we need to add a set of hypervisor agnostic time >keeping guidelines to the documentation. I just wanted to make sure >there wasn't anything KVM specific that should be added as well. > >Thanks, >-John > >On May 15, 2013, at 12:48 PM, Marcus Sorensen wrote: > >> Just the general one that system vms sync their time to the >> hypervisor, thus admins need to keep the hypervisor time correct. It >> sounds like that will be the case for all three, if we can manage it. >> >> On Wed, May 15, 2013 at 10:44 AM, John Burwell >>wrote: >>> Marcus, >>> >>> Excellent. So, it looks like we have KVM resolved. We just need to >>>address Xen and VMWare now. Do you think we need to any guidance to >>>the documentation regarding KVM time keeping (e.g. environmental >>>prerequisites)? >>> >>> Thanks, >>> -John >>> >>> >>> >>> On May 15, 2013, at 12:39 PM, Marcus Sorensen >>>wrote: >>> KVM LibvirtComputingResource has been patched in master. Tested on master, 4.1, and both the acton and current system vm templates. This patch makes system vms use 'kvmclock' for their timer, which is a vm driver that gets it's time from the hypervisor. No change to the system vm template itself. bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 On Wed, May 15, 2013 at 9:08 AM, Chip Childers wrote: > On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: >> Chip, >> >> One other item I neglected to mention was that clock sync, at least >>for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. >>Previously when I encountered these issues, syncing the host's clock >>and rebuilding the system VMs addressed the issue. I assumed, but >>never verified, that the SSVM was syncing back against the host's >>clock through hypervisor. During my testing yesterday, aside from >>hard setting the clock, I was unable to force clock sync on the SSVM. >> >> Thanks, >> -John > > I think that's our issue right now... answering the question: Why is > this only an issue now? Did we just get lucky up to this point? >Since > the SSVMs are the same template as the timeframe you mention, I tend >to > believe that you / we were just lucky. > > Anyone else have thoughts? > >> >> On May 15, 2013, at 10:18 AM, Chip Childers >> wrote: >> >>> Starting a thread on this specific issue. >>> >>> CLOUDSTACK-2492 was opened, which is basically the fact that the >>>System >>> VMs aren't syncing time to the host or to an NTP server. The S3 >>> integration is broken because of this problem, and therefore could >>>not >>> be considered a function available in 4.1 if we release as is. >>> >>> We need input from people that know about the current system VMs >>>(the >>> 3.x VMs), as well as the possibility of using the newer ones that >>>we >>> have been considering experimental for 4.1.0. >>> >>> What should we do? >>> >>> -chip >> >> >>> >
Re: [ACS41] System VMs not syncing time - does this block the release?
Marcus, Agreed. I think we need to add a set of hypervisor agnostic time keeping guidelines to the documentation. I just wanted to make sure there wasn't anything KVM specific that should be added as well. Thanks, -John On May 15, 2013, at 12:48 PM, Marcus Sorensen wrote: > Just the general one that system vms sync their time to the > hypervisor, thus admins need to keep the hypervisor time correct. It > sounds like that will be the case for all three, if we can manage it. > > On Wed, May 15, 2013 at 10:44 AM, John Burwell wrote: >> Marcus, >> >> Excellent. So, it looks like we have KVM resolved. We just need to address >> Xen and VMWare now. Do you think we need to any guidance to the >> documentation regarding KVM time keeping (e.g. environmental prerequisites)? >> >> Thanks, >> -John >> >> >> >> On May 15, 2013, at 12:39 PM, Marcus Sorensen wrote: >> >>> KVM LibvirtComputingResource has been patched in master. Tested on >>> master, 4.1, and both the acton and current system vm templates. This >>> patch makes system vms use 'kvmclock' for their timer, which is a vm >>> driver that gets it's time from the hypervisor. No change to the >>> system vm template itself. >>> >>> bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 >>> >>> On Wed, May 15, 2013 at 9:08 AM, Chip Childers >>> wrote: On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: > Chip, > > One other item I neglected to mention was that clock sync, at least for > Xen system VMs, wasn't an issue in the Jan-Feb timeframe. Previously > when I encountered these issues, syncing the host's clock and rebuilding > the system VMs addressed the issue. I assumed, but never verified, that > the SSVM was syncing back against the host's clock through hypervisor. > During my testing yesterday, aside from hard setting the clock, I was > unable to force clock sync on the SSVM. > > Thanks, > -John I think that's our issue right now... answering the question: Why is this only an issue now? Did we just get lucky up to this point? Since the SSVMs are the same template as the timeframe you mention, I tend to believe that you / we were just lucky. Anyone else have thoughts? > > On May 15, 2013, at 10:18 AM, Chip Childers > wrote: > >> Starting a thread on this specific issue. >> >> CLOUDSTACK-2492 was opened, which is basically the fact that the System >> VMs aren't syncing time to the host or to an NTP server. The S3 >> integration is broken because of this problem, and therefore could not >> be considered a function available in 4.1 if we release as is. >> >> We need input from people that know about the current system VMs (the >> 3.x VMs), as well as the possibility of using the newer ones that we >> have been considering experimental for 4.1.0. >> >> What should we do? >> >> -chip > > >>
Re: [ACS41] System VMs not syncing time - does this block the release?
Just the general one that system vms sync their time to the hypervisor, thus admins need to keep the hypervisor time correct. It sounds like that will be the case for all three, if we can manage it. On Wed, May 15, 2013 at 10:44 AM, John Burwell wrote: > Marcus, > > Excellent. So, it looks like we have KVM resolved. We just need to address > Xen and VMWare now. Do you think we need to any guidance to the > documentation regarding KVM time keeping (e.g. environmental prerequisites)? > > Thanks, > -John > > > > On May 15, 2013, at 12:39 PM, Marcus Sorensen wrote: > >> KVM LibvirtComputingResource has been patched in master. Tested on >> master, 4.1, and both the acton and current system vm templates. This >> patch makes system vms use 'kvmclock' for their timer, which is a vm >> driver that gets it's time from the hypervisor. No change to the >> system vm template itself. >> >> bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 >> >> On Wed, May 15, 2013 at 9:08 AM, Chip Childers >> wrote: >>> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: Chip, One other item I neglected to mention was that clock sync, at least for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. Previously when I encountered these issues, syncing the host's clock and rebuilding the system VMs addressed the issue. I assumed, but never verified, that the SSVM was syncing back against the host's clock through hypervisor. During my testing yesterday, aside from hard setting the clock, I was unable to force clock sync on the SSVM. Thanks, -John >>> >>> I think that's our issue right now... answering the question: Why is >>> this only an issue now? Did we just get lucky up to this point? Since >>> the SSVMs are the same template as the timeframe you mention, I tend to >>> believe that you / we were just lucky. >>> >>> Anyone else have thoughts? >>> On May 15, 2013, at 10:18 AM, Chip Childers wrote: > Starting a thread on this specific issue. > > CLOUDSTACK-2492 was opened, which is basically the fact that the System > VMs aren't syncing time to the host or to an NTP server. The S3 > integration is broken because of this problem, and therefore could not > be considered a function available in 4.1 if we release as is. > > We need input from people that know about the current system VMs (the > 3.x VMs), as well as the possibility of using the newer ones that we > have been considering experimental for 4.1.0. > > What should we do? > > -chip >
Re: [ACS41] System VMs not syncing time - does this block the release?
Marcus, Linux seems like reasonable assumption to me for the 4.1 and 4.2 releases because there doesn't appear a way to easily override the images. I think we need to reconsider how we provision system VMs post-4.2 to address a number of issues. If/when we address those issues, it become an issue. Thanks, -John On May 15, 2013, at 12:40 PM, Marcus Sorensen wrote: > Note that this means system vms are required to be linux, as > 'kvmclock' doesn't work on other OSes. I haven't seen anyone proposing > Windows or BSD system vms, so this seemed safe, but perhaps something > to keep in mind. > > On Wed, May 15, 2013 at 10:39 AM, Marcus Sorensen wrote: >> KVM LibvirtComputingResource has been patched in master. Tested on >> master, 4.1, and both the acton and current system vm templates. This >> patch makes system vms use 'kvmclock' for their timer, which is a vm >> driver that gets it's time from the hypervisor. No change to the >> system vm template itself. >> >> bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 >> >> On Wed, May 15, 2013 at 9:08 AM, Chip Childers >> wrote: >>> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: Chip, One other item I neglected to mention was that clock sync, at least for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. Previously when I encountered these issues, syncing the host's clock and rebuilding the system VMs addressed the issue. I assumed, but never verified, that the SSVM was syncing back against the host's clock through hypervisor. During my testing yesterday, aside from hard setting the clock, I was unable to force clock sync on the SSVM. Thanks, -John >>> >>> I think that's our issue right now... answering the question: Why is >>> this only an issue now? Did we just get lucky up to this point? Since >>> the SSVMs are the same template as the timeframe you mention, I tend to >>> believe that you / we were just lucky. >>> >>> Anyone else have thoughts? >>> On May 15, 2013, at 10:18 AM, Chip Childers wrote: > Starting a thread on this specific issue. > > CLOUDSTACK-2492 was opened, which is basically the fact that the System > VMs aren't syncing time to the host or to an NTP server. The S3 > integration is broken because of this problem, and therefore could not > be considered a function available in 4.1 if we release as is. > > We need input from people that know about the current system VMs (the > 3.x VMs), as well as the possibility of using the newer ones that we > have been considering experimental for 4.1.0. > > What should we do? > > -chip
Re: [ACS41] System VMs not syncing time - does this block the release?
Marcus, Excellent. So, it looks like we have KVM resolved. We just need to address Xen and VMWare now. Do you think we need to any guidance to the documentation regarding KVM time keeping (e.g. environmental prerequisites)? Thanks, -John On May 15, 2013, at 12:39 PM, Marcus Sorensen wrote: > KVM LibvirtComputingResource has been patched in master. Tested on > master, 4.1, and both the acton and current system vm templates. This > patch makes system vms use 'kvmclock' for their timer, which is a vm > driver that gets it's time from the hypervisor. No change to the > system vm template itself. > > bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 > > On Wed, May 15, 2013 at 9:08 AM, Chip Childers > wrote: >> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: >>> Chip, >>> >>> One other item I neglected to mention was that clock sync, at least for Xen >>> system VMs, wasn't an issue in the Jan-Feb timeframe. Previously when I >>> encountered these issues, syncing the host's clock and rebuilding the >>> system VMs addressed the issue. I assumed, but never verified, that the >>> SSVM was syncing back against the host's clock through hypervisor. During >>> my testing yesterday, aside from hard setting the clock, I was unable to >>> force clock sync on the SSVM. >>> >>> Thanks, >>> -John >> >> I think that's our issue right now... answering the question: Why is >> this only an issue now? Did we just get lucky up to this point? Since >> the SSVMs are the same template as the timeframe you mention, I tend to >> believe that you / we were just lucky. >> >> Anyone else have thoughts? >> >>> >>> On May 15, 2013, at 10:18 AM, Chip Childers >>> wrote: >>> Starting a thread on this specific issue. CLOUDSTACK-2492 was opened, which is basically the fact that the System VMs aren't syncing time to the host or to an NTP server. The S3 integration is broken because of this problem, and therefore could not be considered a function available in 4.1 if we release as is. We need input from people that know about the current system VMs (the 3.x VMs), as well as the possibility of using the newer ones that we have been considering experimental for 4.1.0. What should we do? -chip >>> >>>
Re: [ACS41] System VMs not syncing time - does this block the release?
Note that this means system vms are required to be linux, as 'kvmclock' doesn't work on other OSes. I haven't seen anyone proposing Windows or BSD system vms, so this seemed safe, but perhaps something to keep in mind. On Wed, May 15, 2013 at 10:39 AM, Marcus Sorensen wrote: > KVM LibvirtComputingResource has been patched in master. Tested on > master, 4.1, and both the acton and current system vm templates. This > patch makes system vms use 'kvmclock' for their timer, which is a vm > driver that gets it's time from the hypervisor. No change to the > system vm template itself. > > bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 > > On Wed, May 15, 2013 at 9:08 AM, Chip Childers > wrote: >> On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: >>> Chip, >>> >>> One other item I neglected to mention was that clock sync, at least for Xen >>> system VMs, wasn't an issue in the Jan-Feb timeframe. Previously when I >>> encountered these issues, syncing the host's clock and rebuilding the >>> system VMs addressed the issue. I assumed, but never verified, that the >>> SSVM was syncing back against the host's clock through hypervisor. During >>> my testing yesterday, aside from hard setting the clock, I was unable to >>> force clock sync on the SSVM. >>> >>> Thanks, >>> -John >> >> I think that's our issue right now... answering the question: Why is >> this only an issue now? Did we just get lucky up to this point? Since >> the SSVMs are the same template as the timeframe you mention, I tend to >> believe that you / we were just lucky. >> >> Anyone else have thoughts? >> >>> >>> On May 15, 2013, at 10:18 AM, Chip Childers >>> wrote: >>> >>> > Starting a thread on this specific issue. >>> > >>> > CLOUDSTACK-2492 was opened, which is basically the fact that the System >>> > VMs aren't syncing time to the host or to an NTP server. The S3 >>> > integration is broken because of this problem, and therefore could not >>> > be considered a function available in 4.1 if we release as is. >>> > >>> > We need input from people that know about the current system VMs (the >>> > 3.x VMs), as well as the possibility of using the newer ones that we >>> > have been considering experimental for 4.1.0. >>> > >>> > What should we do? >>> > >>> > -chip >>> >>>
Re: [ACS41] System VMs not syncing time - does this block the release?
KVM LibvirtComputingResource has been patched in master. Tested on master, 4.1, and both the acton and current system vm templates. This patch makes system vms use 'kvmclock' for their timer, which is a vm driver that gets it's time from the hypervisor. No change to the system vm template itself. bfc5887a1bf6b41e88dd7a8f9987fcee8d3d9175 On Wed, May 15, 2013 at 9:08 AM, Chip Childers wrote: > On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: >> Chip, >> >> One other item I neglected to mention was that clock sync, at least for Xen >> system VMs, wasn't an issue in the Jan-Feb timeframe. Previously when I >> encountered these issues, syncing the host's clock and rebuilding the system >> VMs addressed the issue. I assumed, but never verified, that the SSVM was >> syncing back against the host's clock through hypervisor. During my testing >> yesterday, aside from hard setting the clock, I was unable to force clock >> sync on the SSVM. >> >> Thanks, >> -John > > I think that's our issue right now... answering the question: Why is > this only an issue now? Did we just get lucky up to this point? Since > the SSVMs are the same template as the timeframe you mention, I tend to > believe that you / we were just lucky. > > Anyone else have thoughts? > >> >> On May 15, 2013, at 10:18 AM, Chip Childers >> wrote: >> >> > Starting a thread on this specific issue. >> > >> > CLOUDSTACK-2492 was opened, which is basically the fact that the System >> > VMs aren't syncing time to the host or to an NTP server. The S3 >> > integration is broken because of this problem, and therefore could not >> > be considered a function available in 4.1 if we release as is. >> > >> > We need input from people that know about the current system VMs (the >> > 3.x VMs), as well as the possibility of using the newer ones that we >> > have been considering experimental for 4.1.0. >> > >> > What should we do? >> > >> > -chip >> >>
Re: [ACS41] System VMs not syncing time - does this block the release?
On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: > Chip, > > One other item I neglected to mention was that clock sync, at least for Xen > system VMs, wasn't an issue in the Jan-Feb timeframe. Previously when I > encountered these issues, syncing the host's clock and rebuilding the system > VMs addressed the issue. I assumed, but never verified, that the SSVM was > syncing back against the host's clock through hypervisor. During my testing > yesterday, aside from hard setting the clock, I was unable to force clock > sync on the SSVM. > > Thanks, > -John I think that's our issue right now... answering the question: Why is this only an issue now? Did we just get lucky up to this point? Since the SSVMs are the same template as the timeframe you mention, I tend to believe that you / we were just lucky. Anyone else have thoughts? > > On May 15, 2013, at 10:18 AM, Chip Childers wrote: > > > Starting a thread on this specific issue. > > > > CLOUDSTACK-2492 was opened, which is basically the fact that the System > > VMs aren't syncing time to the host or to an NTP server. The S3 > > integration is broken because of this problem, and therefore could not > > be considered a function available in 4.1 if we release as is. > > > > We need input from people that know about the current system VMs (the > > 3.x VMs), as well as the possibility of using the newer ones that we > > have been considering experimental for 4.1.0. > > > > What should we do? > > > > -chip > >
Re: [ACS41] System VMs not syncing time - does this block the release?
Chip, One other item I neglected to mention was that clock sync, at least for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. Previously when I encountered these issues, syncing the host's clock and rebuilding the system VMs addressed the issue. I assumed, but never verified, that the SSVM was syncing back against the host's clock through hypervisor. During my testing yesterday, aside from hard setting the clock, I was unable to force clock sync on the SSVM. Thanks, -John On May 15, 2013, at 10:18 AM, Chip Childers wrote: > Starting a thread on this specific issue. > > CLOUDSTACK-2492 was opened, which is basically the fact that the System > VMs aren't syncing time to the host or to an NTP server. The S3 > integration is broken because of this problem, and therefore could not > be considered a function available in 4.1 if we release as is. > > We need input from people that know about the current system VMs (the > 3.x VMs), as well as the possibility of using the newer ones that we > have been considering experimental for 4.1.0. > > What should we do? > > -chip
Re: [ACS41] System VMs not syncing time - does this block the release?
On Wed, May 15, 2013 at 10:28:01AM -0400, John Burwell wrote: > Chip, > > The issues with clock drift on the system VMs goes farther and deeper than > S3-backed Secondary Storage. Essentially, anything the system vms do that > involves time can not be trusted. For example, the timestamps of files > written by the SSVM. Bear in mind that it is possible for a system vm to > have a slow clock. Therefore, in a worst case scenario, the timestamp of > the file would be in the past breaking any logic that scans for updated > files. Additionally, correlating logs on a system VM with the management > server or other parts of the infrastructure is difficult to impossible in > these types of clock drift scenarios. In summary, when time in a > distributed system gets skewed, there are a raft of subtle but significant > operational issues that emerge. > > It is also important to note that fixing this issue is larger than simply > running NTP on the system VMs. As I noted on the ticket, each hypervisor > has a recommended approach for ensuring clock synchronization (e.g. VMWare > and KVM provide daemons and/or kernel drivers to sync clocks properly). > The proper fix for the issue will be to implement those best practices in > each hypervisor-specific system VM ISO. I think the biggest challenge to > implementing the fix will be testing more than development. Well clarified, thanks John! > > Thanks, > -John > > > > On Wed, May 15, 2013 at 10:18 AM, Chip Childers > wrote: > > > Starting a thread on this specific issue. > > > > CLOUDSTACK-2492 was opened, which is basically the fact that the System > > VMs aren't syncing time to the host or to an NTP server. The S3 > > integration is broken because of this problem, and therefore could not > > be considered a function available in 4.1 if we release as is. > > > > We need input from people that know about the current system VMs (the > > 3.x VMs), as well as the possibility of using the newer ones that we > > have been considering experimental for 4.1.0. > > > > What should we do? > > > > -chip > >
Re: [ACS41] System VMs not syncing time - does this block the release?
Chip, The issues with clock drift on the system VMs goes farther and deeper than S3-backed Secondary Storage. Essentially, anything the system vms do that involves time can not be trusted. For example, the timestamps of files written by the SSVM. Bear in mind that it is possible for a system vm to have a slow clock. Therefore, in a worst case scenario, the timestamp of the file would be in the past breaking any logic that scans for updated files. Additionally, correlating logs on a system VM with the management server or other parts of the infrastructure is difficult to impossible in these types of clock drift scenarios. In summary, when time in a distributed system gets skewed, there are a raft of subtle but significant operational issues that emerge. It is also important to note that fixing this issue is larger than simply running NTP on the system VMs. As I noted on the ticket, each hypervisor has a recommended approach for ensuring clock synchronization (e.g. VMWare and KVM provide daemons and/or kernel drivers to sync clocks properly). The proper fix for the issue will be to implement those best practices in each hypervisor-specific system VM ISO. I think the biggest challenge to implementing the fix will be testing more than development. Thanks, -John On Wed, May 15, 2013 at 10:18 AM, Chip Childers wrote: > Starting a thread on this specific issue. > > CLOUDSTACK-2492 was opened, which is basically the fact that the System > VMs aren't syncing time to the host or to an NTP server. The S3 > integration is broken because of this problem, and therefore could not > be considered a function available in 4.1 if we release as is. > > We need input from people that know about the current system VMs (the > 3.x VMs), as well as the possibility of using the newer ones that we > have been considering experimental for 4.1.0. > > What should we do? > > -chip >
