On Tuesday, September 16, 2025 at 3:53:22 AM UTC-4 Nikolaus Rath wrote:
On Tue, 16 Sep 2025, at 01:29, '[email protected]' via s3ql wrote: On Monday, September 15, 2025 at 12:49:16 PM UTC-4 Nikolaus Rath wrote: On Mon, 15 Sep 2025, at 15:53, '[email protected]' via s3ql wrote: On Monday, September 15, 2025 at 4:05:32 AM UTC-4 Nikolaus Rath wrote: On Mon, 15 Sep 2025, at 02:36, '[email protected]' via s3ql wrote: > Running fsck.s3ql --force on this filesystem takes about 4 days, is there > a way to avoid doing this? Which steps take the longest? Depending on that, --fast may or may not help. The current fsck seems to be taking about the same time as the previous one, the last one was using s3ql 5.3 with the --fast option, and the longest steps were: Checking DB integrity, approx 1 day Nothing that can be done here, unfortunately. This is just SQLite doing its thing. Only thing that might help is a faster CPU or putting the DB on a faster disk. Uploading metadata, approx 3 days That's only after an unclean unmount, right? It should be much faster if you run fsck.s3ql --force on a clean filesystem? The problem is that fsck.s3ql needs to re-upload the entire database (109 GB in your case), because the information about which parts were modified was lost when mount.s3ql crashed. Is the upload time (3 days for 109 GB, maybe 50 GB after compression) consistent with your bandwidth? If bandwidth is the limiting factor, then in theory this could be sped up by calculating a checksum of each DB block, comparing it with the checksum of the block stored in remote storage, and only re-uploading if the checksum changed. That'd require someone to write the code, of course :-). Bandwidth does not seem to be the limiting factor, I'm getting roughly 100KB/sec upload to google storage, but I get 5MB/sec or more copying data to/from other machines. In fact, nothing on the machine seems maxed out, top says cpu usage less than 20% and memory usage less than 2%, iotop does not see that much disk usage, so not sure where the bottleneck is... If you truly uploaded at 100 KB/sec, then it would take you about (50*1024^3) / (100*1024) / (60*60) = 145 hours to upload 50 GB. So either compression is a lot better than I'd expect, or you must be uploading more quickly than that. What's your metadata block size (specified on mkfs.s3ql time)? I suspect it's relatively small, and since metadata is uploaded sequentially, you're limited by network latency rather than bandwidth. This will be addressed by https://github.com/s3ql/s3ql/issues/310, which I plan to work on if ever I find some free time... Best, -Nikolaus Not sure what's happening with bandwidth, maybe there are spikes of usage I'm missing... I don't know what the metadata block size is I'm afraid :( It looks like I made this filesystem in Sep 2021 using s3ql 3.7.3 and bash history just says mkfs.s3ql gs://bucketname/... Joseph -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/s3ql/d869d1ef-5386-4caa-977e-193cba15eb67n%40googlegroups.com.
