Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-22 Thread Marcus Sorensen
Isn't there some other way we can sync the time as a triage that would avoid a large test on the system vm template? Like adding a time flag to one of the commonly-run system vm scripts (or even just the ones that trigger tasks that require accurate time), which would run a 'date' command in the

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-20 Thread Chip Childers
On Wed, May 15, 2013 at 01:11:53PM -0700, Chiradeep Vittal wrote: For VMWare, the command vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a patch for /etc/init.d/cloud-early-config to enable it Please do! -chip

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-20 Thread Chip Childers
On Mon, May 20, 2013 at 3:59 PM, Chip Childers chip.child...@sungard.com wrote: On Wed, May 15, 2013 at 01:11:53PM -0700, Chiradeep Vittal wrote: For VMWare, the command vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a patch for /etc/init.d/cloud-early-config to enable it

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-20 Thread Chip Childers
We appear to be at an impasse here as well. I'm going to start a VOTE on this issue, given that we have no easy answer. On Wed, May 15, 2013 at 02:49:41PM -0700, Chiradeep Vittal wrote: Well, I disagree, from the perspective of hundreds of production clouds. No harm has been perceived in

[ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chip Childers
Starting a thread on this specific issue. CLOUDSTACK-2492 was opened, which is basically the fact that the System VMs aren't syncing time to the host or to an NTP server. The S3 integration is broken because of this problem, and therefore could not be considered a function available in 4.1 if we

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread John Burwell
Chip, One other item I neglected to mention was that clock sync, at least for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. Previously when I encountered these issues, syncing the host's clock and rebuilding the system VMs addressed the issue. I assumed, but never verified, that

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chip Childers
On Wed, May 15, 2013 at 11:03:16AM -0400, John Burwell wrote: Chip, One other item I neglected to mention was that clock sync, at least for Xen system VMs, wasn't an issue in the Jan-Feb timeframe. Previously when I encountered these issues, syncing the host's clock and rebuilding the

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread John Burwell
Marcus, Excellent. So, it looks like we have KVM resolved. We just need to address Xen and VMWare now. Do you think we need to any guidance to the documentation regarding KVM time keeping (e.g. environmental prerequisites)? Thanks, -John On May 15, 2013, at 12:39 PM, Marcus Sorensen

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread John Burwell
Marcus, Linux seems like reasonable assumption to me for the 4.1 and 4.2 releases because there doesn't appear a way to easily override the images. I think we need to reconsider how we provision system VMs post-4.2 to address a number of issues. If/when we address those issues, it become an

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chiradeep Vittal
According to our resident Xen expert, any PV kernel automatically syncs to the hardware clock on dom0. On 5/15/13 9:50 AM, John Burwell jburw...@basho.com wrote: Marcus, Agreed. I think we need to add a set of hypervisor agnostic time keeping guidelines to the documentation. I just wanted to

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chiradeep Vittal
Perhaps this is a problem with DevCloud? http://nerdboys.com/2011/03/15/how-to-fix-virtualbox-time-synchonization-pr oblems/ On 5/15/13 10:17 AM, Chiradeep Vittal chiradeep.vit...@citrix.com wrote: According to our resident Xen expert, any PV kernel automatically syncs to the hardware clock on

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread John Burwell
Chiradeep, The issue I am experiencing is that the system VMs are not syncing to dom0 on devcloud (i.e. the dom0 clock and the SSVM clock are different). As I mentioned earlier in this thread, the syncing was working previously which seems to jibe with your findings. What mechanism is used to

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chiradeep Vittal
The previous ones were on XS 5.6 FP2 This one's on XS 6.0.2 r-275166-VM 18:22:10 up 8 days, domU: Wed May 15 18:22:10 UTC 2013 dom0: Wed May 15 11:22:10 PDT 2013 On 5/15/13 11:22 AM, Chiradeep Vittal chiradeep.vit...@citrix.com wrote: The normal S3 time sync is 15 minutes. I can't imagine a

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread John Burwell
Chiradeep, As I mentioned earlier, this issue is larger than S3-backed Secondary Storage. It just happens that this issue was surfaced by testing that feature.Clock drift exceeding than a few seconds can be operational issue (e.g. file timestamps, logging, etc). A lack of reliable clock

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chip Childers
On Wed, May 15, 2013 at 02:48:13PM -0400, John Burwell wrote: All of these things being said, it appears that the Xen behavior may be a regression that can be addressed with a relatively straightforward fix (dropping the proper file in /proc/sys/xen), Agreed. Anyone want to submit the

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chiradeep Vittal
/proc/sys is not a regular filesystem and cannot be added to from the shell. Drivers need to add nodes into this filesystem. On 5/15/13 11:50 AM, Chip Childers chip.child...@sungard.com wrote: On Wed, May 15, 2013 at 02:48:13PM -0400, John Burwell wrote: All of these things being said, it

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chip Childers
On Wed, May 15, 2013 at 12:02:41PM -0700, Chiradeep Vittal wrote: /proc/sys is not a regular filesystem and cannot be added to from the shell. Drivers need to add nodes into this filesystem. Backing up a bit... This is the current system VM image that the 4.1 software should be using on Xen:

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chiradeep Vittal
I am not sure why it is missing, but I will refer to Citrix XenServer 6.0 Virtual Machine Installation Guide http://s.apache.org/YGn And quote Time Handling in Linux VMs By default, the clocks in a Linux VM are synchronized to the clock running on the control domain, and cannot be independently

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chiradeep Vittal
For VMWare, the command vmware-toolbox-cmd timesync status returns 'Disabled'. I can submit a patch for /etc/init.d/cloud-early-config to enable it On 5/15/13 12:23 PM, Chiradeep Vittal chiradeep.vit...@citrix.com wrote: I am not sure why it is missing, but I will refer to Citrix XenServer 6.0

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chiradeep Vittal
Did some further digging around as to why /proc/sys/xen/independent_wallclock is not there on the Debian system vm. TLDR; the kernel is PvOps (I.e., just a regular kernel that works like a PV kernel, not a specialized paravirt kernel). To eliminate special-casing, the independent_wallclock

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread John Burwell
Chiradeep, As I think thought it, I don't think a documentation note will sufficient because the SSVM can be destroyed and respawned by CloudStack without intervention by a human. Therefore, we can get into a situation where an operator installs and configures NTP, and then at some point in

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread John Burwell
All, One other point to consider is that we only want NTP running for Xen system VMs. Running NTP and VMWare Tools together causes big problems because the two fight each other to sync the clock. I don;t know about KVM, but I wouldn't be surprised if it doesn't have same types of problems.

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chiradeep Vittal
Sure, I agree. But I'd also point out that for the vast majority of CloudStack users (4.1 at least), this is not going to be an issue. I suggest deferring this to 4.1.1 A new template (easy or not) does require a full regression QA round. On 5/15/13 2:07 PM, John Burwell jburw...@basho.com wrote:

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread John Burwell
Chiradeep, I disagree regarding the impact of this issue. Anyone running an SSVM will be affected by this issue because clocks will eventually drift (sooner rather than later) and when they do, any timestamps rendered by a system VM will unreliable (e.g. files creation and modified times, log

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Chiradeep Vittal
Well, I disagree, from the perspective of hundreds of production clouds. No harm has been perceived in those clouds due to this defect. If they can live with it for several years, then they can live with it for a few more months. On 5/15/13 2:35 PM, Wido den Hollander w...@widodh.nl wrote: On

Re: [ACS41] System VMs not syncing time - does this block the release?

2013-05-15 Thread Marcus Sorensen
Nothing wrong with running NTP in KVM System VM btw. Wido I think they are right that it will be fighting with the hypervisor synced clock. It might not really pose much of an issue technically, though.