I’m using JClouds BlobStore in an application that’s using it to send files to
S3. It does this in batches like after a snapshot is run on a database. And
then all of the files are transferred to S3. We were testing error states.
And we were testing changing a User’s Access credentials from inactive to
active and back. When we go from active to inactive credentials during this
transfer to S3. Our thread gets killed. The last log entry is from JClouds.
Our code is literally catching all errors, and so nothing is being thrown at
that time. Our application is in clojure. You can see in the log, that we
upload successfully fine, and then after we switch to invalid we “correct” get
errors that get returned to our application. However, then we hit the SEVERE
log line, and then our thread is dead and never comes back :-(
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,398 Sending message
'["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Data.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,496 Uploading file:
snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-CompressionInfo.db
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,498 Throttle value requested is
out of range, using default.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:14,695 Sending message
'["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-CompressionInfo.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,797 Uploading file:
snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-98-Index.db
DEBUG [async-dispatch-31] 2014-12-05 16:29:14,809 Throttle value requested is
out of range, using default.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,105 Sending message
'["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-98-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,204 Sending message
'["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Statistics.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,305 Sending message
'["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-96-Index.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,406 Sending message
'["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"sstable","state":"running","details":{"filename":"OpsCenter-rollups300-jb-97-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,503 Got exception HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
opsagent.backups.entities.SSTable@48fee98c to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at
org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at
opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at
opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at
clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at
clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at
clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at
clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Summary.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,504 Sending message
'["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Summary.db
HTTP\/1.1 -> HTTP\/1.1 403
Forbidden","filename":"OpsCenter-rollups300-jb-96-Summary.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,600 Got exception HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
opsagent.backups.entities.SSTable@91afeb42 to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at
org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at
opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at
opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at
clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at
clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at
clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at
clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-rollups300-jb-96-Filter.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,601 Sending message
'["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-rollups300-jb-96-Filter.db
HTTP\/1.1 -> HTTP\/1.1 403
Forbidden","filename":"OpsCenter-rollups300-jb-96-Filter.db","keyspace":"OpsCenter","cf":"rollups300"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,696 Got exception HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
opsagent.backups.entities.SSTable@a4c93 to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at
org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at
opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at
opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at
clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at
clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at
clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at
clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,697 Sending message
'["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-CompressionInfo.db
HTTP\/1.1 -> HTTP\/1.1 403
Forbidden","filename":"OpsCenter-pdps-jb-114-CompressionInfo.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,792 Got exception HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
opsagent.backups.entities.SSTable@44e5d97 to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at
org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at
opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at
opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at
clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at
clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at
clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at
clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Statistics.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,793 Sending message
'["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Statistics.db
HTTP\/1.1 -> HTTP\/1.1 403
Forbidden","filename":"OpsCenter-pdps-jb-114-Statistics.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
ERROR [async-dispatch-31] 2014-12-05 16:29:15,888 Got exception HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden while sending
opsagent.backups.entities.SSTable@8ca4473b to :380a2db2ac2042c5bd43043d8ae53b6f
org.jclouds.rest.AuthorizationException: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
HTTP/1.1 -> HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122)
at
org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:94)
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:135)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.invoke(InvokeSyncToAsyncHttpMethod.java:128)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:94)
at
org.jclouds.rest.internal.InvokeSyncToAsyncHttpMethod.apply(InvokeSyncToAsyncHttpMethod.java:55)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy48.objectExists(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.blobExists(S3BlobStore.java:173)
at org.jclouds.blobstore2$blob_exists_QMARK_.invoke(blobstore2.clj:238)
at opsagent.backups.destinations$create_blob.invoke(destinations.clj:43)
at
opsagent.backups.destinations$eval15431$fn__15460.invoke(destinations.clj:123)
at
opsagent.backups.destinations$eval15244$fn__15245$G__15235__15254.invoke(destinations.clj:19)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655$fn__15683.invoke(staging.clj:46)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650$fn__15655.invoke(staging.clj:44)
at
opsagent.backups.staging$start_staging_BANG_$fn__15649$state_machine__13248__auto____15650.invoke(staging.clj:40)
at
clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)
at
clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)
at
clojure.core.async.impl.ioc_macros$take_BANG_$fn__13264.invoke(ioc_macros.clj:953)
at
clojure.core.async.impl.channels.ManyToManyChannel$fn__8952.invoke(channels.clj:133)
at clojure.lang.AFn.run(AFn.java:24)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.jclouds.http.HttpResponseException: request: HEAD
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-114-Data.db
HTTP/1.1 failed with response: HTTP/1.1 403 Forbidden
at
org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63)
... 25 more
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:15,890 Sending message
'["172.31.10.206",{"details":{"error":"HEAD
https:\/\/phalliday-opscenter-backups-4.s3.amazonaws.com\/snapshots\/728139a7-093b-40f7-96a9-767a178fa5a5\/sstables\/OpsCenter-pdps-jb-114-Data.db
HTTP\/1.1 -> HTTP\/1.1 403
Forbidden","filename":"OpsCenter-pdps-jb-114-Data.db","keyspace":"OpsCenter","cf":"pdps"},"state":"running","type":"sstable","destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [node-details-1] 2014-12-05 16:29:16,030 Starting real-time collection
DEBUG [node-details-1] 2014-12-05 16:29:16,030 Collecting real-time data
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,034 Sending message
'["172.31.10.206",{"os-load":0.07}]' to channel '/node-details'.
DEBUG [clojure-agent-send-off-pool-9] 2014-12-05 16:29:16,298 Sending message
'["172.31.10.206",{"destination_id":"380a2db2ac2042c5bd43043d8ae53b6f","type":"schema","state":"running","details":{"keyspace":"OpsCenter","filename":"OpsCenter\/schema.json"},"parent_req":"eb335075-765a-46b0-bcb8-89188d93cb1d"}]'
to channel '/snapshot/status'.
DEBUG [async-dispatch-31] 2014-12-05 16:29:16,402 Uploading file:
snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
DEBUG [async-dispatch-31] 2014-12-05 16:29:16,404 Throttle value requested is
out of range, using default.
ERROR [async-dispatch-31] 2014-12-05 16:29:16,510 Dec 05, 2014 4:29:16 PM
org.jclouds.logging.jdk.JDKLogger logError
SEVERE: Cannot retry after server error, command is not replayable:
[method=org.jclouds.aws.s3.AWSS3AsyncClient.public abstract
com.google.common.util.concurrent.ListenableFuture
org.jclouds.s3.S3AsyncClient.putObject(java.lang.String,org.jclouds.s3.domain.S3Object,org.jclouds.s3.options.PutObjectOptions[])[phalliday-opscenter-backups-4,
[metadata=[key=snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db,
bucket=null, uri=null, eTag=null, cacheControl=null,
contentMetadata=[contentDisposition=null, contentEncoding=null,
contentLanguage=null, contentLength=107, contentMD5=null,
contentType=application/octet-stream, expires=null], lastModified=null,
owner=null, storageClass=STANDARD, userMetadata={}]],
[Lorg.jclouds.s3.options.PutObjectOptions;@4e0ecbfe], request=PUT
https://phalliday-opscenter-backups-4.s3.amazonaws.com/snapshots/728139a7-093b-40f7-96a9-767a178fa5a5/sstables/OpsCenter-pdps-jb-115-Summary.db
HTTP/1.1]
DEBUG [node-details-1] 2014-12-05 16:29:21,034 Starting real-time collection
DEBUG [node-details-1] 2014-12-05 16:29:21,035 Collecting real-time data
DEBUG [node-details-1] 2014-12-05 16:29:21,038 Send requested, but nothing new
to send
<http://www.datastax.com/>
PETER HALLIDAY
OpsCenter | Sr Software Engineer | 607.329.6905 | [email protected]
<https://twitter.com/phalliday>
<https://www.linkedin.com/profile/view?id=7030852>
