[s3ql] guidance needed - 10TB in selfhosted distributed filesystem

[Marcin Ciesielski]

Hi, need some advice

I store more than 10TB photos in S3QL on a self hosted distributed 
filesystem.
Technically it is LizardFS, so S3QL does not talk to it natively but when 
mounted locally its POSIX compliant.

I wonder what is the best approach for the fastest backups.
Option 1. I mount LizardFS locally and setup S3QL local://
Option2. I mount LizardFS on one of my remote servers and expose via minio 
and then do S3QL s3://
Option3. like Option2 but I do S3QL local:// on my remote server and expose 
S3QL filesystem via minio.

What are the main differences between the three approaches - speed wise?
We are talking home broadband around 10mbps so everything will be slow but 
maybe one will be less slow?

I assume Option3 will push the most data through since deduplication 
happens remotely, something like rclone could help here but I think it 
still will be the slowest.

I believe the choice is between local vs S3/minio

What are your opinions?

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[s3ql] Re: guidance needed - 10TB in selfhosted distributed filesystem

[Marcin Ciesielski]

it's a SOHO setup I try to manage in my limited spare time,
Multiple data centres and erasure coding 4.2 for resiliency (and price).

Ceph is way more complex and as far as I remember does erasure coding in 
one datacenter only. Other than that only replication.
EdgeFS requires kubernetes (I gave up on setup due to complexity, forget to 
manage this) and I think it does not do erasure.
Minio would be an option but I cannot easily add disk space, and I am not 
prepared to pay in advance for too much.

Lizard is not perfect by any means, but it's a trade-off.

Would use of Option 2 (S3 api) have a speed benefit over going local://
technically it's one layer more but maybe S3QL handles S3 backend somehow 
different/better?

Isaac suggested that possibly S3QL writes object metadata in the header of 
the written object when using local:// so when s3ql head a object or need 
to fsck actually need to open the file, for S3 Metadata is written 
separately of the object. @Nikratio  can give 
us more info about this.

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[s3ql] ConstraintError: UNIQUE constraint failed: obj_ids.id

[Marcin Ciesielski]

Version S3QL 3.1
stored on local

My filesystem run out of space while writing to S3QL.
Now during fsck I am getting this error


Starting fsck of local:///mnt/.s3qldata/
Using cached metadata.
WARNING: Remote metadata is outdated.
Checking DB integrity...
Creating temporary extra indices...
Checking lost+found...
Checking for dirty cache objects...
Checking names (refcounts)...
Checking contents (names)...
Checking contents (inodes)...
Checking contents (parent inodes)...
Checking for temporary objects (backend)...
Checking objects (reference counts)...
Checking objects (backend)...
..processed 692625 objects so far..
Dropping temporary indices...
ERROR: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/local/bin/fsck.s3ql", line 11, in 
load_entry_point('s3ql==3.1', 'console_scripts', 'fsck.s3ql')()
  File 
"/usr/local/lib/python3.5/dist-packages/s3ql-3.1-py3.5-linux-x86_64.egg/s3ql/fsck.py",
 
line 1273, in main
fsck.check(check_cache)
  File 
"/usr/local/lib/python3.5/dist-packages/s3ql-3.1-py3.5-linux-x86_64.egg/s3ql/fsck.py",
 
line 92, in check
self.check_objects_id()
  File 
"/usr/local/lib/python3.5/dist-packages/s3ql-3.1-py3.5-linux-x86_64.egg/s3ql/fsck.py",
 
line 965, in check_objects_id
self.conn.execute('INSERT INTO obj_ids VALUES(?)', (obj_id,))
  File 
"/usr/local/lib/python3.5/dist-packages/s3ql-3.1-py3.5-linux-x86_64.egg/s3ql/database.py",
 
line 98, in execute
self.conn.cursor().execute(*a, **kw)
  File "src/cursor.c", line 236, in resetcursor
apsw.ConstraintError: ConstraintError: UNIQUE constraint failed: obj_ids.id

I cleaned up cache but it did not help
any chance to get it running?

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/6dd31e8e-3847-407c-bd78-68b8ce0f4032%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[s3ql] Re: ConstraintError: UNIQUE constraint failed: obj_ids.id

[Marcin Ciesielski]

Thanks, Nikolaus

I managed to recover. 
It seems that the problem was that I misspelt the cache directory name and 
was pointing into different ones for mount and fsck.
I cleaned the mount one but fsck was probably holding some few weeks old 
data.

Best
Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/b26c20ef-d52b-413e-b6b2-c703f51e276f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[s3ql] [question] how to fix corrutped files returned by verify

[Marcin Ciesielski]

Hi,

is there a way to fix corrupted files returned by s3ql_verify?
I still might have the source files that were backed up to s3ql.


Thank you

Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/b87a777e-bb5b-4bb2-99ab-46592a828875%40googlegroups.com.

Re: [s3ql] [question] how to fix corrutped files returned by verify

[Marcin Ciesielski]

Thank, you, 

that's exactly what I was looking for.
I didn't find it before, somehow google search was of much less help than 
you

Cheers
Marcin

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/d71b5763-f276-4275-b6ee-07e8faac2896%40googlegroups.com.

Re: [s3ql] Changing backend providers via bucket copy - feasible?

[Marcin Ciesielski]

On Wednesday, 13 May 2020 15:42:09 UTC+1, Daniel Jagszent wrote:
>
> Hello Xomex, 
> > [...] Is it feasible to copy my data at the bucket level from provider 
> > A to B (using a 3rd party data copying service) and then connect s3ql 
> > to the new data location.[...] 
> You need to use https://www.rath.org/s3ql-docs/contrib.html#clone-fs-py 
> to copy a S3QL file system from one storage backend to a different 
> storage backend. Only copying the data from one bucket to another does 
> not work. S3QL uses backend specific metadata (e.g. S3 metadata is a 
> little bit different than Openstack Swift metadata). 
>
> Unfortunately clone-fs.py does not do server-to-server copies. You might 
> be better off to run clone-fs.py inside a throw away VM with a decent 
> internet connection. 
>
>
Do you guys know if clone-fs.py can pickup where interrupted? Or basically 
copy across what was not yet copied like rclone or rsync.


 

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/f407dffb-45fa-43d1-9cde-bacb0336e477%40googlegroups.com.

Re: [s3ql] Error when compiling s3ql 3.5.0

[Marcin Ciesielski]

I have the same problem, any ideas?

On Thursday, 16 July 2020 22:23:14 UTC+1, Alexandre Gonçalves wrote:
>
> Alexandre Gonçalves > escreveu 
> no dia quinta, 16/07/2020 à(s) 18:02:
>
>> Daniel Jagszent > escreveu no dia 
>> quinta, 16/07/2020 à(s) 15:04:
>>
>>> Hi,
>>>
>>> [...]
>>> Installing lib/libfuse3.so.3.9.2 to 
>>> /usr/local/lib/x86_64-linux-gnu/libfuse3.so.3.9.2
>>> [...]
>>>
>>> is the directory /usr/local/lib/x86_64-linux-gnu in your library search 
>>> path?
>>> You may need to (a) install libfuse into a standard path of your OS or 
>>> (b) add the path to /etc/ld/so.conf and run ldconfig
>>> See https://linux.die.net/man/8/ldconfig 
>>>
>>>
>>>
>> That was it! After adding the path of libfuse lib using ldconfig, 
>> everything went smoothly!
>>
>> Thanks.
>>
>>  
>>
>
> After successfully building s3ql, and passed the tests, I'm seeing these 
> errors: 
>
> Traceback (most recent call last):
>   File "/usr/local/bin/mount.s3ql", line 33, in 
> sys.exit(load_entry_point('s3ql==3.5.0', 'console_scripts', 
> 'mount.s3ql')())
>   File "/usr/local/bin/mount.s3ql", line 25, in importlib_load_entry_point
> return next(matches).load()
>   File 
> "/usr/local/lib/python3.6/dist-packages/importlib_metadata/__init__.py", 
> line 105, in load
> module = import_module(match.group('module'))
>   File "/usr/lib/python3.6/importlib/__init__.py", line 126, in 
> import_module
> return _bootstrap._gcd_import(name[level:], package, level)
>   File "", line 994, in _gcd_import
>   File "", line 971, in _find_and_load
>   File "", line 955, in 
> _find_and_load_unlocked
>   File "", line 656, in _load_unlocked
>   File "", line 626, in 
> _load_backward_compatible
>   File 
> "/usr/local/lib/python3.6/dist-packages/s3ql-3.5.0-py3.6-linux-x86_64.egg/s3ql/mount.py",
>  
> line 18, in 
> from .metadata import (download_metadata, upload_metadata, 
> dump_and_upload_metadata,
>   File 
> "/usr/local/lib/python3.6/dist-packages/s3ql-3.5.0-py3.6-linux-x86_64.egg/s3ql/metadata.py",
>  
> line 13, in 
> from .deltadump import INTEGER, BLOB, dump_table, load_table
> ImportError: cannot import name 'INTEGER'
>
> Any suggestions?
>
> Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/854db7c6-5243-449f-9f93-e54129413d81o%40googlegroups.com.

Re: [s3ql] Error when compiling s3ql 3.5.0

[Marcin Ciesielski]

I am sorry I think I must have answered via email and it git lost
Yes I do precisely this command
The I also execute testing and it passes without issues

Buster
Python 3.6

On Tuesday, 21 July 2020 17:38:42 UTC+1, Daniel Jagszent wrote:
>
>
> I compile from the release archive
> I can see the deltadump.c file
> however, I am still getting this[...]
>
> do you execute "python3 setup.py build_ext  --inplace" (or something 
> similar)?
>

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/552bed5e-fde1-4c07-b538-4e0f91376cdeo%40googlegroups.com.

Re: [s3ql] Error when compiling s3ql 3.5.0

[Marcin Ciesielski]

this is precisely the error I am getting
building docker image from python:3.6-slim - hence working as root

Traceback (most recent call last):
  File "/root/.local/bin/mkfs.s3ql", line 33, in 
sys.exit(load_entry_point('s3ql==3.5.0', 'console_scripts', 'mkfs.s3ql'
)())
  File "/root/.local/bin/mkfs.s3ql", line 25, in importlib_load_entry_point
return next(matches).load()
  File 
"/root/.local/lib/python3.6/site-packages/importlib_metadata/__init__.py", 
line 105, in load
module = import_module(match.group('module'))
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in 
import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 994, in _gcd_import
  File "", line 971, in _find_and_load
  File "", line 955, in _find_and_load_unlocked
  File "", line 656, in _load_unlocked
  File "", line 626, in 
_load_backward_compatible
  File 
"/root/.local/lib/python3.6/site-packages/s3ql-3.5.0-py3.6-linux-x86_64.egg/s3ql/mkfs.py"
, line 15, in 
from .metadata import dump_and_upload_metadata, create_tables
  File 
"/root/.local/lib/python3.6/site-packages/s3ql-3.5.0-py3.6-linux-x86_64.egg/s3ql/metadata.py"
, line 13, in 
from .deltadump import INTEGER, BLOB, dump_table, load_table
ImportError: cannot import name 'INTEGER'




On Tuesday, 21 July 2020 17:38:42 UTC+1, Daniel Jagszent wrote:
>
>
> I compile from the release archive
> I can see the deltadump.c file
> however, I am still getting this[...]
>
> do you execute "python3 setup.py build_ext  --inplace" (or something 
> similar)?
>

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/a5c6ddde-f38e-4d02-90ad-63e1499e90e3o%40googlegroups.com.

Re: [s3ql] Error when compiling s3ql 3.5.0

[Marcin Ciesielski]

plus I get this error regardless if working from git repo or from release 
archive

On Tuesday, 21 July 2020 17:38:42 UTC+1, Daniel Jagszent wrote:
>
>
> I compile from the release archive
> I can see the deltadump.c file
> however, I am still getting this[...]
>
> do you execute "python3 setup.py build_ext  --inplace" (or something 
> similar)?
>

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/f96f2a30-b51a-48c4-ae1f-716efd2d792do%40googlegroups.com.

Re: [s3ql] Error when compiling s3ql 3.5.0

[Marcin Ciesielski]

I finally was able to avoid this issue using fedora image

On Monday, 27 July 2020 22:26:51 UTC+1, r0ps3c wrote:
>
> I am seeing the same issue, and have been seeing it since at least 3.4.1. 
> Looking at setup.py's history, looks like the cython build code hasn't been 
> touched, and likely hasn't been updated, in a while.
>
> For reference, I've been able to reproduce this on (docker-ised) ubuntu 
> 20.04.1 and alpine 3.12.
>
> On Monday, July 27, 2020 at 4:44:49 PM UTC-4 ciesiels...@gmail.com wrote:
>
>> plus I get this error regardless if working from git repo or from release 
>> archive
>>
>>
>> On Tuesday, 21 July 2020 17:38:42 UTC+1, Daniel Jagszent wrote:
>>
>>>
>>> I compile from the release archive
>>> I can see the deltadump.c file
>>> however, I am still getting this[...]
>>>
>>> do you execute "python3 setup.py build_ext  --inplace" (or something 
>>> similar)?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/ef4d0491-0f08-4b80-b1c3-0d3755bb0f6do%40googlegroups.com.

[s3ql] Re: [ANNOUNCE] New S3QL pre-release available

[Marcin Ciesielski]

@Nikolaus
So it seems that S3QL is still alive :)

On Wednesday, 17 May 2023 at 09:43:19 UTC+1 Nikolaus Rath wrote:

> Hi all,
>
> I have recently started working on S3QL again and am excited to announce
> a pre-release of S3QL 5.0!
>
> There's been a large number of internal cleanups, but the most important
> change is that:
>
> S3QL (finally!) no longer limits the compressed metadata size to 5 GB
> because it no longer maintains entire filesystem metadata in a single
> storage object.
>
> Instead, the database file is distributed across multiple backend
> objects with a block size configured at mkfs time. This means that S3QL
> also no longer needs to upload the entire metadata object on unmount;
> and there is no longer a size limit on the metadata.
>
> It would be great if people could give this version a spin, but note
> that there may still be bugs.
>
> The pre-release is available for download from 
> https://github.com/s3ql/s3ql/releases/tag/release-5.0.0-pre1
>
>
> Best,
> -Nikolaus
>
> -- 
> GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>

-- 
You received this message because you are subscribed to the Google Groups 
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to s3ql+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/s3ql/1a71df34-4c0b-42f1-a4c3-2a96fc26184an%40googlegroups.com.