Hello, I use s3ql for quite some time as my apache webroot, which serves about 300gb of files and some php applications. I had few problems with high I/O scenarios in the past (using the 1.x series), where the mount daemon would die under high load, but managed to solve them all with fsck.s3ql after rebooting.
My current server was running s3ql 2.11 on ubuntu 14.04. Earlier today, I was performing full backups from that node to another s3 bucket using duplicity, and after lots of gbs transferred, s3ql failed and I had to reboot the server. Now fsck.s3ql is hanging at ..processed 99500 objects so far.. fsck.log (with --debug) shows a lot of HEAD requests, It seems to be running a verify for every block (about 1400000). I also tried downloading a metadata backup, but it hangs at the same point, here's an excerpt of the ongoing log: 2015-02-16 22:44:39.697 19239:MainThread s3ql.backends.s3c.lookup: lookup(s3ql_data_417399) 2015-02-16 22:44:39.697 19239:MainThread s3ql.backends.s3c._do_request: preparing HEAD /homes3ql_data_417399?None, qs=None 2015-02-16 22:44:39.697 19239:MainThread s3ql.backends.s3c._send_request: _send_request(): HEAD /homes3ql_data_417399 2015-02-16 22:44:39.717 19239:MainThread s3ql.backends.s3c.lookup: lookup(s3ql_data_417400) 2015-02-16 22:44:39.717 19239:MainThread s3ql.backends.s3c._do_request: preparing HEAD /homes3ql_data_417400?None, qs=None 2015-02-16 22:44:39.717 19239:MainThread s3ql.backends.s3c._send_request: _send_request(): HEAD /homes3ql_data_417400 2015-02-16 22:44:39.750 19239:MainThread s3ql.backends.s3c.lookup: lookup(s3ql_data_417401) 2015-02-16 22:44:39.750 19239:MainThread s3ql.backends.s3c._do_request: preparing HEAD /homes3ql_data_417401?None, qs=None 2015-02-16 22:44:39.750 19239:MainThread s3ql.backends.s3c._send_request: _send_request(): HEAD /homes3ql_data_417401 2015-02-16 22:44:39.797 19239:MainThread s3ql.backends.s3c.lookup: lookup(s3ql_data_417402) 2015-02-16 22:44:39.797 19239:MainThread s3ql.backends.s3c._do_request: preparing HEAD /homes3ql_data_417402?None, qs=None 2015-02-16 22:44:39.798 19239:MainThread s3ql.backends.s3c._send_request: _send_request(): HEAD /homes3ql_data_417402 2015-02-16 22:44:39.830 19239:MainThread s3ql.backends.s3c.lookup: lookup(s3ql_data_417403) 2015-02-16 22:44:39.830 19239:MainThread s3ql.backends.s3c._do_request: preparing HEAD /homes3ql_data_417403?None, qs=None Is there any way to disable this behaviour (verifying every block on fsck) ? I'm recovering those files from my backups, but hope to access this volume as well. fsck will probably run for about 11 hours, will post the results then. Thanks in advance Guilherme -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
