[s3ql] Question about data and file deletion timing
I have a backup that involves mounting an s3ql file system and using 'rsync --delete ...' to do the backup. If a large folder (or file) is moved, there is a risk that the files will be 'deleted' before they area created again, possibly resulting in remote data being deleted then recreated. Is this concern genuine? Would doing: 'rsync' (with no --delete) followed by 'rsync --delete ...' be the best way to alleviate the risk of wasted bandwidth (and time) here? Or should I consider a snapshot, the 'rsync --delete' then delete the snapshot? Any advice appreciated! -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/145ce093-4cad-4454-b5d7-ac1c313ff02bn%40googlegroups.com.
Re: [s3ql] Question about data and file deletion timing
lol...that works...didn't know it existed. Thank you! On Tuesday, January 5, 2021 at 7:09:25 PM UTC+11 Cliff Stanford wrote: > On 05/01/2021 05:42, rabidmuta...@gmail.com wrote: > > > If a large folder (or file) is moved, there is a risk that the files > > will be 'deleted' before they area created again, possibly resulting in > > remote data being deleted then recreated. > > How about using `rsync --delete-after` to ensure that all blocks remain > until the transfer is complete? > > Cliff. > > -- > Cliff Stanford > London: +44 20 0222 1666 <+44%2020%200222%201666> Swansea: +44 1792 469666 > <+44%201792%20469666> > Spain: +34 603 777 666 <+34%20603%2077%2076%2066> Estonia: +372 5308 9666 > <+372%205308%209666> > UK Mobile: +44 7973 616 666 <+44%207973%2061> > > -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/11e78102-0833-4117-bb36-44c6966d4432n%40googlegroups.com.
[s3ql] Enforce python3 version?
This is a little sad, but my host has pythion3.6 as the default python, and I use python3.11 for the build. However, all of the build scripts use `#$! ... python3` which starts 3.6. Is there any way that I can get the build process to either set he python version in the "#!" for each file (3.11) or use the python version it was built with? Otherwise, I guess I just edit all the python files in bin/, but that seems like a bad solution. -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/afc78fc0-d4c4-44ba-ba96-4c2dfb02e93an%40googlegroups.com.
Re: [s3ql] Enforce python3 version?
"python3.11 setup.py [...]“ is exactly what I do, but the built files in bin/ have: ``` ~/s3ql/s3ql-5.1.3 # egrep "^#!" bin/* bin/fsck.s3ql:#!/usr/bin/env python3 bin/mkfs.s3ql:#!/usr/bin/env python3 bin/mount.s3ql:#!/usr/bin/env python3 bin/s3ql_oauth_client:#!/usr/bin/env python3 bin/s3ql_verify:#!/usr/bin/env python3 bin/s3qladm:#!/usr/bin/env python3 bin/s3qlcp:#!/usr/bin/env python3 bin/s3qlctrl:#!/usr/bin/env python3 bin/s3qllock:#!/usr/bin/env python3 bin/s3qlrm:#!/usr/bin/env python3 bin/s3qlstat:#!/usr/bin/env python3 bin/umount.s3ql:#!/usr/bin/env python3 ``` On Thursday, April 18, 2024 at 6:36:31 PM UTC+10 Nikolaus Rath wrote: > Hello, > > Which "build scripts" do you mean? You can just call "python3.x setup.py > [...]“ explicitly, and the files that this installs should use the same > interpreter automatically. > > Best, > Nikolaus > > On Thu, 18 Apr 2024, at 07:42, rabidmuta...@gmail.com wrote: > > This is a little sad, but my host has pythion3.6 as the default python, > and I use python3.11 for the build. > > However, all of the build scripts use `#$! ... python3` which starts 3.6. > > Is there any way that I can get the build process to either set he python > version in the "#!" for each file (3.11) or use the python version it was > built with? > > Otherwise, I guess I just edit all the python files in bin/, but that > seems like a bad solution. > > > -- > You received this message because you are subscribed to the Google Groups > "s3ql" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to s3ql+uns...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/s3ql/afc78fc0-d4c4-44ba-ba96-4c2dfb02e93an%40googlegroups.com > > <https://groups.google.com/d/msgid/s3ql/afc78fc0-d4c4-44ba-ba96-4c2dfb02e93an%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to s3ql+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/s3ql/b4f40505-39bc-4d9f-a812-382362d7db76n%40googlegroups.com.
Re: [s3ql] DatabaseChecksumError on fsck
Thank you for this insight! Despite appearances, I am/was running 5.1.2.
Your suggesting of using a later version resolved the problem: simply
running fsck from 5.1.3 worked without issues. Next step is to upgrade to
latest version.
Thanks for the suggestion.
On Saturday, January 11, 2025 at 1:35:30 AM UTC+11 Daniel Jagszent wrote:
> Hello,
> Am 10.01.25 um 07:13 schrieb rabidmuta...@gmail.com:
>
> Corrupt remote data; what is the recommended recovery procedure?
>
> I found this:
> https://groups.google.com/g/s3ql/c/4LM5-5uXfQY/m/6amCKocRAQAJ
>
> ...is it the best approach?
>
> fsck.s3ql
> Starting fsck of
> Using cached metadata.
> File system was not unmounted cleanly
> Checking DB integrity...
> Verifying consistency of most recent metadata backups:
> Checking backup 00becb (from 2025-01-06 00:46:32))...
> Downloaded 2031/17560 metadata blocks (11%)
> Downloaded 17560/17560 metadata blocks (100%)
> ERROR: Uncaught top-level exception:
> Traceback (most recent call last):
> File "/usr/bin/fsck.s3ql", line 33, in
> sys.exit(load_entry_point('s3ql==5.1.3', 'console_scripts',
> 'fsck.s3ql')())
>
> ^
> File
> "/usr/lib64/python3.11/site-packages/s3ql-5.1.2-py3.11-linux-x86_64.egg/s3ql/fsck.py",
>
> line 1342, in main
> verify_metadata_snapshots(backend, count=5,
> include_most_recent=check_current_metadata)
> File
> "/usr/lib64/python3.11/site-packages/s3ql-5.1.2-py3.11-linux-x86_64.egg/s3ql/fsck.py",
>
> line 1435, in verify_metadata_snapshots
> conn = download_metadata(backend, fh.name, params)
>^^^
> File
> "/usr/lib64/python3.11/site-packages/s3ql-5.1.2-py3.11-linux-x86_64.egg/s3ql/database.py",
>
> line 543, in download_metadata
> raise DatabaseChecksumError(db_file, params.db_md5, digest)
> s3ql.database.DatabaseChecksumError: File /tmp/tmp20kx_gk_ has checksum
> 933548326db2ae62d820c2b703cae18c9dda8f7d063f53773f869e58c5c6c1fc, expected
> a4ee0e2c03c34c17341e2d1fe0213a9c614d3a715111c9ce3b0eb04fb3c5a595
>
>
> Since your local database (SQLite file) looks ok ("Checking DB
> integrity..." line without error message), ignoring these remote database
> backup checksum mismatches is a valid route. You should find out why this
> happened, though. Were you affected by
> https://github.com/s3ql/s3ql/issues/321 ? (Did you use S3QL Version <
> 5.1.1 at 2025-01-06 and is your S3QL cache directory a symlink?) Also, do
> not forget to add five valid remote database backups and revert your patch
> afterward, as Henry explained.
>
> Which version of S3QL are you using, btw. The exception indicates a
> strange mix of version 5.1.3 and version 5.1.2. If it is indeed 5.1.2,
> updating to the newest version (5.2.3) might be all that is necessary. Only
> from version 5.1.3 upwards, fsck.s3ql skips unclean metadata backups since
> they will most likely produce those database checksum errors. All 5.x
> versions of S3QL use the same database structure; updating to the latest
> version should be relatively hassle-free, and a downgrade is possible (but
> not advised).
>
>
>
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to s3ql+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/s3ql/8463cf4c-be2d-4e7e-b4bf-0dbf2f0efb4cn%40googlegroups.com.
[s3ql] DatabaseChecksumError on fsck
Corrupt remote data; what is the recommended recovery procedure?
I found this: https://groups.google.com/g/s3ql/c/4LM5-5uXfQY/m/6amCKocRAQAJ
...is it the best approach?
fsck.s3ql
Starting fsck of
Using cached metadata.
File system was not unmounted cleanly
Checking DB integrity...
Verifying consistency of most recent metadata backups:
Checking backup 00becb (from 2025-01-06 00:46:32))...
Downloaded 2031/17560 metadata blocks (11%)
Downloaded 17560/17560 metadata blocks (100%)
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
File "/usr/bin/fsck.s3ql", line 33, in
sys.exit(load_entry_point('s3ql==5.1.3', 'console_scripts',
'fsck.s3ql')())
^
File
"/usr/lib64/python3.11/site-packages/s3ql-5.1.2-py3.11-linux-x86_64.egg/s3ql/fsck.py",
line 1342, in main
verify_metadata_snapshots(backend, count=5,
include_most_recent=check_current_metadata)
File
"/usr/lib64/python3.11/site-packages/s3ql-5.1.2-py3.11-linux-x86_64.egg/s3ql/fsck.py",
line 1435, in verify_metadata_snapshots
conn = download_metadata(backend, fh.name, params)
^^^
File
"/usr/lib64/python3.11/site-packages/s3ql-5.1.2-py3.11-linux-x86_64.egg/s3ql/database.py",
line 543, in download_metadata
raise DatabaseChecksumError(db_file, params.db_md5, digest)
s3ql.database.DatabaseChecksumError: File /tmp/tmp20kx_gk_ has checksum
933548326db2ae62d820c2b703cae18c9dda8f7d063f53773f869e58c5c6c1fc, expected
a4ee0e2c03c34c17341e2d1fe0213a9c614d3a715111c9ce3b0eb04fb3c5a595
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to s3ql+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/s3ql/7e6047b6-d910-4c18-87ce-8353364fa80an%40googlegroups.com.
