On May 14, 2013, at 7:37 PM, Chip Childers <chip.child...@sungard.com> wrote:
> On Tuesday, May 14, 2013, John Burwell wrote: > >> Chip, >> >> After some further discussion about this issue on IRC, Alex and I >> determined that system VM clock drift issue not only breaks S3, but has >> other significant impacts that merit it being a blocker for 4.1 (e.g. >> timestamps of files written by the SSVM being incorrect, log file >> correlation difficulties, sensitivity of other storage protocols to time >> sync, etc). Based on this conversation, I have opened >> CLOUDSTACK-2492<https://issues.apache.org/jira/browse/CLOUDSTACK-2492> to >> capture the issue and track resolution. >> >> Thanks, >> -John >> > > Well shoot. Issuing a new system VM is going to require a bunch of testing > if it's not done as a hand rolled edit of the existing system VMs. > > Who can take this? Was this in 4.0.x ? If yes then maybe we can live with it till 4.2 ? > > >> >> On May 14, 2013, at 6:09 PM, John Burwell >> <jburw...@basho.com<javascript:_e({}, 'cvml', 'jburw...@basho.com');>> >> wrote: >> >> Chip, >> >> The source of the problem appears to be clock drift between the SSVM and >> S3 per following stack trace: >> >> 2013-05-14 06:51:55,400 DEBUG [cloud.utils.S3Utils] >> (agentRequest-Handler-3:) Putting directory >> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5 in >> S3 bucket jsb-cloudstack-templates. >> 2013-05-14 06:51:55,401 DEBUG [cloud.utils.S3Utils] >> (agentRequest-Handler-3:) Creating S3 client with configuration: [protocol: >> https, connectionTimeOut: 50000, maxErrorRetry: 3, socketTimeout: 50000] >> 2013-05-14 06:51:55,403 DEBUG >> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) >> Determining key using account id 1 and template id 5 >> 2013-05-14 06:51:55,403 DEBUG [cloud.utils.S3Utils] >> (agentRequest-Handler-3:) Putting file >> /mnt/SecStorage/93fd0cb0-033b-3248-bcd0-ef6d460635ef/template/tmpl/1/5/template.properties >> into bucket jsb-cloudstack-templates with key >> template/tmpl/1/5/template.properties. >> 2013-05-14 06:51:55,578 ERROR >> [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:) >> Failed to upload template id 5 >> Status Code: 403, AWS Service: Amazon S3, AWS Request ID: >> 970A274E132A9ACB, AWS Error Code: RequestTimeTooSkewed, AWS Error Message: >> The difference between the request time and the current time is too large., >> S3 Extended Request ID: >> 9w8a6YBxTn+WlBg96s9stxWuuP8oQ7ksZtg6++wVRHJfE2qmucrilhoEJVetJui4 >> at >> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:609) >> at >> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:309) >> at >> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:164) >> at >> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2863) >> at >> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1100) >> at >> com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:963) >> at com.cloud.utils.S3Utils.putDirectory(S3Utils.java:282) >> at >> com.cloud.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:414) >> at >> com.cloud.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:212) >> at com.cloud.agent.Agent.processRequest(Agent.java:525) >> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852) >> at com.cloud.utils.nio.Task.run(Task.java:83) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> It does not appear that the System VM image has NTP nor does it seem like >> a valid assumption that an SSVM would have the connectivity to reach an NTP >> server. Therefore, do we have some other means to sync the clock on the >> SSVM (e.g. with the host through hypervisor extensions)? In my previous >> tests, the SSVM seemed to be syncing its time to the host which addressed >> the problem. >> >> Thanks, >> -John >> >> On May 14, 2013, at 4:58 PM, John Burwell >> <jburw...@basho.com<javascript:_e({}, 'cvml', 'jburw...@basho.com');>> >> wrote: >> >> Chip, >> >> I am looking into the issue now. There is a failure when the S3 upload >> template command is issued. I working to determine whether or not the >> cause is environmental or code. >> >> Thanks, >> -John >> >> On May 14, 2013, at 4:56 PM, Chip Childers >> <chip.child...@sungard.com<javascript:_e({}, 'cvml', >> 'chip.child...@sungard.com');>> >> wrote: >> >> Hi all, >> >> We have a clear bug list (blockers and critical) for 4.1.0. I'm going >> to cut a new release candidate tonight. >> >> If there are *any* outstanding issues known, now's the time to raise >> them. >> >> (I'm specifically looking for an ACK from jburwell here, since he >> mentioned a possible S3 feature issue. John, if you can please give me >> a heads up on where you stand in the next few hours, I'd appreciate it.) >> >> -chip >> >> >> >> >>
