[Gluster-users] Docker volume plugin

[Italo Maia]

Is calavera/docker-volume-glusterfs still the advised volume plugin to use
with gluster+docker?

-- 
"A arrogância é a arma dos fracos."

===
Me. Italo Moreira Campelo Maia
Co-fundador do Grupo de Usuários Python do Ceará
Secretário ForHacker (fb.com/ForHackerSpace)
Desenvolvedor Full-Stack, Escritor, Empresário, Visionário
-
Meu Livro , Site ,
Blog 
===
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Bandwidth and latency requirements

[Colin Coe]

Apologies for the late reply.

Further to this, if my Linux clients are connecting uing glusterfs-fuse and
I have my volumes defined like this:
dc1srv1:/gv_fileshare dc2srv1:/gv_fileshare dc1srv2:/gv_fileshare
dc2srv2:/gv_fileshare (replica 2)

How do I ensure that clients in dc1 prefer dc1srv1 and dc1srv2 while
clients in dc2 prefer the dc2 servers?

Is it simply a matter of ordering in /etc/fstab?

Thanks


On Fri, Sep 29, 2017 at 2:29 PM, Karan Sandha  wrote:

> It was simple emulation of network packets on the port of the server node
> using tc tool tc qdisc add dev  root netem delay ms. The
> files were created using dd tool (in-built in linux) and mkdir. Post the
> IO's we verified with no pending heals.
>
>
> Thanks & Regards
>
>
> On Thu, Sep 28, 2017 at 2:06 PM, Arman Khalatyan 
> wrote:
>
>> Interesting table Karan!,
>> Could you please tell us how you did  the benchmark? fio or iozone
>> orsimilar?
>>
>> thanks
>> Arman.
>>
>> On Wed, Sep 27, 2017 at 1:20 PM, Karan Sandha  wrote:
>>
>>> Hi Collin,
>>>
>>> During our arbiter latency testing for completion of ops we found the
>>> below results:-  an arbiter node in another data centre and both the data
>>> bricks in the same data centre,
>>>
>>> 1) File-size 1 KB (1 files )
>>> 2) mkdir
>>>
>>>
>>> Latency
>>>
>>> 5ms
>>>
>>> 10ms
>>>
>>> 20ms
>>>
>>> 50ms
>>>
>>> 100ms
>>>
>>> 200ms
>>>
>>> Ops
>>>
>>> Create
>>>
>>> 755 secs
>>>
>>> 1410 secs
>>>
>>> 2717 secs
>>>
>>> 5874 secs
>>>
>>> 12908 sec
>>>
>>> 26113 sec
>>>
>>> Mkdir
>>>
>>> 922 secs
>>>
>>> 1725 secs
>>>
>>> 3325 secs
>>>
>>> 8127 secs
>>>
>>> 16160 sec
>>>
>>> 30079 sec
>>>
>>>
>>> Thanks & Regards
>>>
>>> On Mon, Sep 25, 2017 at 5:40 AM, Colin Coe  wrote:
>>>
 Hi all

 I've googled but can't find an answer to my question.

 I have two data centers.  Currently, I have a replica (count of 2 plus
 arbiter) in one data center but is used by both.

 I want to change this to be a distributed replica across the two data
 centers.

 There is a 20Mbps pipe and approx 22 ms latency. Is this sufficient?

 I really don't want to do the geo-replication in its current form.

 Thanks

 CC

 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://lists.gluster.org/mailman/listinfo/gluster-users

>>>
>>>
>>>
>>> --
>>>
>>> KARAN SANDHA
>>>
>>> ASSOCIATE QUALITY ENGINEER
>>>
>>> Red Hat Bangalore 
>>>
>>> ksan...@redhat.comM: 9888009555 IM: Karan on @irc
>>> 
>>> TRIED. TESTED. TRUSTED. 
>>> @redhatnews    Red Hat
>>>    Red Hat
>>> 
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
>
> --
>
> KARAN SANDHA
>
> ASSOCIATE QUALITY ENGINEER
>
> Red Hat Bangalore 
>
> ksan...@redhat.comM: 9888009555 IM: Karan on @irc
> 
> TRIED. TESTED. TRUSTED. 
> @redhatnews    Red Hat
>    Red Hat
> 
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Announcing Glusterfs release 3.12.2 (Long Term Maintenance)

[Jiffin Tony Thottan]

The Gluster community is pleased to announce the release of Gluster 
3.12.2 (packages available at [1,2,3]).


Release notes for the release can be found at [4].

We still carry following major issues that is reported in the 
release-notes as follows,


1.) - Expanding a gluster volume that is sharded may cause file corruption

Sharded volumes are typically used for VM images, if such volumes 
are expanded or possibly contracted (i.e add/remove bricks and 
rebalance) there are reports of VM images getting corrupted.


The last known cause for corruption (Bug #1465123) has a fix with 
this release. As further testing is still in progress, the issue is 
retained as a major issue.


Status of this bug can be tracked here, #1465123


2 .) Gluster volume restarts fail if the sub directory export feature is 
in use. Status of this issue can be tracked here, #1501315


3.) Mounting a gluster snapshot will fail, when attempting a FUSE based 
mount of the snapshot. So for the current users, it is recommend to only 
access snapshot via


".snaps" directory on a mounted gluster volume. Status of this issue can 
be tracked here, #1501378


Thanks,
 Gluster community


[1] https://download.gluster.org/pub/gluster/glusterfs/3.12/3.12.2/ 


[2] https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.12
[3] https://build.opensuse.org/project/subprojects/home:glusterfs

[4] Release notes: 
https://gluster.readthedocs.io/en/latest/release-notes/3.12.2/


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] small files performance

[Gandalf Corvotempesta]

Where did you read 2k IOPS?

Each disk is able to do about 75iops as I'm using SATA disk, getting even
closer to 2000 it's impossible

Il 13 ott 2017 9:42 AM, "Szymon Miotk"  ha scritto:

> Depends what you need.
> 2K iops for small file writes is not a bad result.
> In my case I had a system that was just poorly written and it was
> using 300-1000 iops for constant operations and was choking on
> cleanup.
>
>
> On Thu, Oct 12, 2017 at 6:23 PM, Gandalf Corvotempesta
>  wrote:
> > So, even with latest version, gluster is still unusable with small files
> ?
> >
> > 2017-10-12 10:51 GMT+02:00 Szymon Miotk :
> >> I've analyzed small files performance few months ago, because I had
> >> huge performance problems with small files writes on Gluster.
> >> The read performance has been improved in many ways in recent releases
> >> (md-cache, parallel-readdir, hot-tier).
> >> But write performance is more or less the same and you cannot go above
> >> 10K smallfiles create - even with SSD or Optane drives.
> >> Even ramdisk is not helping much here, because the bottleneck is not
> >> in the storage performance.
> >> Key problems I've noticed:
> >> - LOOKUPs are expensive, because there is separate query for every
> >> depth level of destination directory (md-cache helps here a bit,
> >> unless you are creating lot of directories). So the deeper the
> >> directory structure, the worse.
> >> - for every file created, Gluster creates another file in .glusterfs
> >> directory, doubling the required IO and network latency. What's worse,
> >> XFS, the recommended filesystem, doesn't like flat directory sturcture
> >> with thousands files in each directory. But that's exactly how Gluster
> >> stores its metadata in .glusterfs, so the performance decreases by
> >> 40-50% after 10M files.
> >> - complete directory structure is created on each of the bricks. So
> >> every mkdir results in io on every brick you have in the volume.
> >> - hot-tier may be great for improving reads, but for small files
> >> writes it actually kills performance even more.
> >> - FUSE driver requires context switch between userspace and kernel
> >> each time you create a file, so with small files the context switches
> >> are also taking their toll
> >>
> >> The best results I got were:
> >> - create big file on Gluster, mount it as XFS over loopback interface
> >> - 13.5K smallfile writes. Drawback - you can use it only on one
> >> server, as XFS will crash when two servers will write to it.
> >> - use libgfapi - 20K smallfile writes performance. Drawback - no nice
> >> POSIX filesystem, huge CPU usage on Gluster server.
> >>
> >> I was testing with 1KB files, so really small.
> >>
> >> Best regards,
> >> Szymon Miotk
> >>
> >> On Fri, Oct 6, 2017 at 4:43 PM, Gandalf Corvotempesta
> >>  wrote:
> >>> Any update about this?
> >>> I've seen some works about optimizing performance for small files, is
> >>> now gluster "usable" for storing, in example, Maildirs or git sources
> >>> ?
> >>>
> >>> at least in 3.7 (or 3.8, I don't remember exactly), extracting kernel
> >>> sources took about 4-5 minutes.
> >>> ___
> >>> Gluster-users mailing list
> >>> Gluster-users@gluster.org
> >>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] how to verify bitrot signed file manually?

[Amudhan P]

any update?.

why is it marked bad?

Any way to find out what happened to the file?


On Tue, Oct 3, 2017 at 12:44 PM, Amudhan P  wrote:

>
> my volume is distributed disperse volume 8+2 EC.
> file1 and file2 are different files lying in same brick. I am able to read
> the file from mount point without any issue because of EC it reads rest of
> the available blocks in other nodes.
>
> my question is "file1" sha256 value matches bitrot signature value but
> still, it is also marked as bad by scrubber daemon. why is that?
>
>
>
> On Fri, Sep 29, 2017 at 12:52 PM, Kotresh Hiremath Ravishankar <
> khire...@redhat.com> wrote:
>
>> Hi Amudhan,
>>
>> Sorry for the late response as I was busy with other things. You are
>> right bitrot uses sha256 for checksum.
>> If file-1, file-2 are marked bad, the I/O should be errored out with EIO.
>> If that is not happening, we need
>> to look further into it. But what's the file contents of file-1 and
>> file-2 on the replica bricks ? Are they
>> matching ?
>>
>> Thanks and Regards,
>> Kotresh HR
>>
>> On Mon, Sep 25, 2017 at 4:19 PM, Amudhan P  wrote:
>>
>>> resending mail.
>>>
>>>
>>> On Fri, Sep 22, 2017 at 5:30 PM, Amudhan P  wrote:
>>>
 ok, from bitrot code I figured out gluster using sha256 hashing algo.


 Now coming to the problem, during scrub run in my cluster some of my
 files were marked as bad in few set of nodes.
 I just wanted to confirm bad file. so, I have used "sha256sum" tool in
 Linux to manually get file hash.

 here is the result.

 file-1, file-2 marked as bad by scrub and file-3 is healthy.

 file-1 sha256 and bitrot signature value matches but still it's been
 marked as bad.

 file-2 sha256 and bitrot signature value don't match, could be a victim
 of bitrot or bitflip.file is still readable without any issue and no errors
 found in the drive.

 file-3 sha256 and bitrot signature matches and healthy.


 file-1 output from

 "sha256sum" = "71eada9352b1352aaef0f806d3d56
 1768ce2df905ded1668f665e06eca2d0bd4"


 "getfattr -m. -e hex -d "
 # file: file-1
 trusted.bit-rot.bad-file=0x3100
 trusted.bit-rot.signature=0x01020071eada9352b135
 2aaef0f806d3d561768ce2df905ded1668f665e06eca2d0bd4
 trusted.bit-rot.version=0x020058e4f3b40006793d
 trusted.ec.config=0x080a02000200
 trusted.ec.dirty=0x
 trusted.ec.size=0x000718996701
 trusted.ec.version=0x00038c4c00038c4d
 trusted.gfid=0xf078a24134fe4f9bb953eca8c28dea9a

 output scrub log:
 [2017-09-02 13:02:20.311160] A [MSGID: 118023]
 [bit-rot-scrub.c:244:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
 CORRUPTION DETECTED: Object /file-1 {Brick: /media/disk16/brick16 | GFID:
 f078a241-34fe-4f9b-b953-eca8c28dea9a}
 [2017-09-02 13:02:20.311579] A [MSGID: 118024]
 [bit-rot-scrub.c:264:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
 Marking /file-1 [GFID: f078a241-34fe-4f9b-b953-eca8c28dea9a | Brick:
 /media/disk16/brick16] as corrupted..

 file-2 output from

 "sha256sum" = "c41ef9c81faed4f3e6010ea67984c
 3cfefd842f98ee342939151f9250972dcda"


 "getfattr -m. -e hex -d "
 # file: file-2
 trusted.bit-rot.bad-file=0x3100
 trusted.bit-rot.signature=0x0102009162cb17d4f0be
 e676fcb7830c5286d05b8e8940d14f3d117cb90b7b1defc129
 trusted.bit-rot.version=0x020058e4f3b400019bb2
 trusted.ec.config=0x080a02000200
 trusted.ec.dirty=0x
 trusted.ec.size=0x403433f6
 trusted.ec.version=0x201a201b
 trusted.gfid=0xa50012b0a632477c99232313928d239a

 output scrub log:
 [2017-09-02 05:18:14.003156] A [MSGID: 118023]
 [bit-rot-scrub.c:244:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
 CORRUPTION DETECTED: Object /file-2 {Brick: /media/disk13/brick13 | GFID:
 a50012b0-a632-477c-9923-2313928d239a}
 [2017-09-02 05:18:14.006629] A [MSGID: 118024]
 [bit-rot-scrub.c:264:bitd_compare_ckum] 0-qubevaultdr-bit-rot-0:
 Marking /file-2 [GFID: a50012b0-a632-477c-9923-2313928d239a | Brick:
 /media/disk13/brick13] as corrupted..


 file-3 output from

 "sha256sum" = "a590735b3c8936cc7ca9835128a19
 c38a3f79c8fd53fddc031a9349b7e273f27"


 "getfattr -m. -e hex -d "
 # file: file-3
 trusted.bit-rot.signature=0x010200a590735b3c8936
 cc7ca9835128a19c38a3f79c8fd53fddc031a9349b7e273f27
 trusted.bit-rot.version=0x020058e4f3b400019bb2
 trusted.ec.config=0x080a02000200
 trusted.ec.dirty=0x
 trusted.ec.size=0x3530fc96
 trusted.ec.version=0x1a981a99
 trusted.gfid=0x10d8920e42cd42cf9448b8bf3941c192



 most of the bitrot bad files ar