On Sun, 30 Nov 2014, Nikolaus Rath wrote:
On 11/30/2014 12:50 AM, Shannon Dealy wrote:
I suspect this is because you copied the objects into a new bucket, but
did not include the associated metadata.
Which tool did you use to do the copy, and how did you call it?
I used the AWS command line tools:
aws s3 sync s3://src-bucket s3://dest-bucket
It did fail part way through the transfer, so I restarted it with the
same command.
When you use the S3 Management Console
(https://console.aws.amazon.com/s3/home) to look at the "s3ql_metadata"
object in the original bucket and the copy, does it have the same
metadata?
While the s3ql_metadata file is there, in the new bucket it is missing
all of the keys except the Content-Type. Not sure whether this means
the fsck without cached data trashed it, or if amazon's sync was
trashing the transfered data because I used it incorrectly or it is
broken somehow.
I think you have to blame the aws tool for this one. If you think
fsck.s3ql is at fault, that's easy enough to check: just run the sync
command again and see if the metadata is present before running fsck.s3ql.
I don't know if that's a bug in aws, or if you're using it incorrectly,
but "gsutil" is able to copy buckets including metadata (in case you
want to try that).
Poking around, it appears that the metadata was lost only on the
s3ql_metadataXXXX files, so I deleted them and ran the "aws s3 sync"
command again. The new files again were missing the metadata, so I just
copied the 13 s3ql_metadataXXXX files using the aws console, and this time
the metadata was preserved.
I didn't see anything special about these files with regard to permissions
or other information, is there anything special about how these files
are created/stored on S3? If not, then possibly simply having "metadata"
in the filename may cause the aws s3 sync command to lose the file
metadata.
In any case, once I replaced the s3ql_metadata files I was able to run
fsck.s3ql on the copy of the file system bucket without the local data
cache and it resulted in exactly the same failure mode/exception "PRIMARY
KEY must be unique".
Will take a look at gsutil, thanks for the suggestion.
FWIW.
Regards,
Shannon C. Dealy | DeaTech Research Inc.
de...@deatech.com | - Custom Software Development -
USA Phone: +1 800-467-5820 | - Natural Building Instruction -
numbers : +1 541-929-4089 | www.deatech.com
--
To UNSUBSCRIBE, email to debian-bugs-rc-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
