Re: [Gluster-devel] [Gluster-users] Feature Request: Lock Volume Settings

2016-11-13 Thread David Gossage 
On Sun, Nov 13, 2016 at 6:35 PM, Lindsay Mathieson <
lindsay.mathie...@gmail.com> wrote:

> As discussed recently, it is way to easy to make destructive changes
> to a volume,e.g change shard size. This can corrupt the data with no
> warnings and its all to easy to make a typo or access the wrong volume
> when doing 3am maintenance ...
>
> So I'd like to suggest something like the following:
>
>   gluster volume lock 
>
> Setting this would fail all:
> - setting changes
> - add bricks
> - remove bricks
> - delete volume
>
>   gluster volume unlock 
>
> would allow all changes to be made.
>
> Just a thought, open to alternate suggestions.
>
> Thanks
>
> +
sounds handy

> --
> Lindsay
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-08-03 Thread David Gossage 
On Wed, Aug 3, 2016 at 7:57 AM, Lindsay Mathieson <
lindsay.mathie...@gmail.com> wrote:

> On 3/08/2016 10:45 PM, Lindsay Mathieson wrote:
>
> On 3/08/2016 2:26 PM, Krutika Dhananjay wrote:
>
> Once I deleted old content from test volume it mounted to oVirt via
> storage add when previously it would error out.  I am now creating a test
> VM with default disk caching settings (pretty sure oVirt is defaulting to
> none rather than writeback/through).  So far all shards are being created
> properly.
>
>
> I can confirm that it works with ProxMox VM's in direct (no cache mode) as
> well.
>
>
> Also Gluster 3.8.1 is good to
>

ugh almost done updating to 3.7.14 and already feeling the urge to start
testing and updating to 3.8 branch.

>
> --
> Lindsay Mathieson
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] GlusterFS-3.7.14 released

2016-08-02 Thread David Gossage 
On Tue, Aug 2, 2016 at 6:01 AM, Lindsay Mathieson <
lindsay.mathie...@gmail.com> wrote:

> On 2/08/2016 5:07 PM, Kaushal M wrote:
>
>> GlusterFS-3.7.14 has been released. This is a regular minor release.
>> The release-notes are available at
>>
>> https://github.com/gluster/glusterfs/blob/release-3.7/doc/release-notes/3.7.14.md
>>
>
> Thanks Kaushal, I'll check it out
>
>
So far on my test box its working as expected.  At least the issues that
prevented it from running as before have disappeared.  Will need to see how
my test VM behaves after a few days.



-- 
> Lindsay Mathieson
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-08-02 Thread David Gossage 
So far both dd commands that failed previously worked fine on 3.7.14

Once I deleted old content from test volume it mounted to oVirt via storage
add when previously it would error out.  I am now creating a test VM with
default disk caching settings (pretty sure oVirt is defaulting to none
rather than writeback/through).  So far all shards are being created
properly.

Load is sky rocketing but I have all 3 gluster bricks running off 1 hard
drive on test box so I would expect horrible io/load issues with that.

Very promising so far.  Thank you developers for your help in working
through this.

Once I have the VM installed and running will test for a few days and make
sure it doesn't have any freeze or locking issues then will roll this out
to working cluster.



*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284

On Wed, Jul 27, 2016 at 8:37 AM, David Gossage 
wrote:

> On Tue, Jul 26, 2016 at 9:38 PM, Krutika Dhananjay 
> wrote:
>
>> Yes please, could you file a bug against glusterfs for this issue?
>>
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1360785
>
>
>>
>>
>> -Krutika
>>
>> On Wed, Jul 27, 2016 at 1:39 AM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>> Has a bug report been filed for this issue or should l I create one with
>>> the logs and results provided so far?
>>>
>>> *David Gossage*
>>> *Carousel Checks Inc. | System Administrator*
>>> *Office* 708.613.2284
>>>
>>> On Fri, Jul 22, 2016 at 12:53 PM, David Gossage <
>>> dgoss...@carouselchecks.com> wrote:
>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur 
>>>> wrote:
>>>>
>>>>> On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <
>>>>> samp...@neutraali.net> wrote:
>>>>> > Here is a quick way how to test this:
>>>>> > GlusterFS 3.7.13 volume with default settings with brick on ZFS
>>>>> dataset. gluster-test1 is server and gluster-test2 is client mounting with
>>>>> FUSE.
>>>>> >
>>>>> > Writing file with oflag=direct is not ok:
>>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
>>>>> count=1 bs=1024000
>>>>> > dd: failed to open ‘file’: Invalid argument
>>>>> >
>>>>> > Enable network.remote-dio on Gluster Volume:
>>>>> > [root@gluster-test1 gluster]# gluster volume set gluster
>>>>> network.remote-dio enable
>>>>> > volume set: success
>>>>> >
>>>>> > Writing small file with oflag=direct is ok:
>>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
>>>>> count=1 bs=1024000
>>>>> > 1+0 records in
>>>>> > 1+0 records out
>>>>> > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s
>>>>> >
>>>>> > Writing bigger file with oflag=direct is ok:
>>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
>>>>> count=100 bs=1M
>>>>> > 100+0 records in
>>>>> > 100+0 records out
>>>>> > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s
>>>>> >
>>>>> > Enable Sharding on Gluster Volume:
>>>>> > [root@gluster-test1 gluster]# gluster volume set gluster
>>>>> features.shard enable
>>>>> > volume set: success
>>>>> >
>>>>> > Writing small file  with oflag=direct is ok:
>>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
>>>>> count=1 bs=1M
>>>>> > 1+0 records in
>>>>> > 1+0 records out
>>>>> > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s
>>>>> >
>>>>> > Writing bigger file with oflag=direct is not ok:
>>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
>>>>> count=100 bs=1M
>>>>> > dd: error writing ‘file3’: Operation not permitted
>>>>> > dd: closing output file ‘file3’: Operation not permitted
>>>>> >
>>>>>
>>>>>
>>>>> Thank you for these tests! would it be possible to share the brick and
>>>>> client logs?
>>>>>
>>>>
>>>> Not sure if his tests are same as my setup but here i

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-27 Thread David Gossage 
On Tue, Jul 26, 2016 at 9:38 PM, Krutika Dhananjay 
wrote:

> Yes please, could you file a bug against glusterfs for this issue?
>

https://bugzilla.redhat.com/show_bug.cgi?id=1360785


>
>
> -Krutika
>
> On Wed, Jul 27, 2016 at 1:39 AM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> Has a bug report been filed for this issue or should l I create one with
>> the logs and results provided so far?
>>
>> *David Gossage*
>> *Carousel Checks Inc. | System Administrator*
>> *Office* 708.613.2284
>>
>> On Fri, Jul 22, 2016 at 12:53 PM, David Gossage <
>> dgoss...@carouselchecks.com> wrote:
>>
>>>
>>>
>>>
>>> On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur 
>>> wrote:
>>>
>>>> On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen <
>>>> samp...@neutraali.net> wrote:
>>>> > Here is a quick way how to test this:
>>>> > GlusterFS 3.7.13 volume with default settings with brick on ZFS
>>>> dataset. gluster-test1 is server and gluster-test2 is client mounting with
>>>> FUSE.
>>>> >
>>>> > Writing file with oflag=direct is not ok:
>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
>>>> count=1 bs=1024000
>>>> > dd: failed to open ‘file’: Invalid argument
>>>> >
>>>> > Enable network.remote-dio on Gluster Volume:
>>>> > [root@gluster-test1 gluster]# gluster volume set gluster
>>>> network.remote-dio enable
>>>> > volume set: success
>>>> >
>>>> > Writing small file with oflag=direct is ok:
>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
>>>> count=1 bs=1024000
>>>> > 1+0 records in
>>>> > 1+0 records out
>>>> > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s
>>>> >
>>>> > Writing bigger file with oflag=direct is ok:
>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
>>>> count=100 bs=1M
>>>> > 100+0 records in
>>>> > 100+0 records out
>>>> > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s
>>>> >
>>>> > Enable Sharding on Gluster Volume:
>>>> > [root@gluster-test1 gluster]# gluster volume set gluster
>>>> features.shard enable
>>>> > volume set: success
>>>> >
>>>> > Writing small file  with oflag=direct is ok:
>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
>>>> count=1 bs=1M
>>>> > 1+0 records in
>>>> > 1+0 records out
>>>> > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s
>>>> >
>>>> > Writing bigger file with oflag=direct is not ok:
>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
>>>> count=100 bs=1M
>>>> > dd: error writing ‘file3’: Operation not permitted
>>>> > dd: closing output file ‘file3’: Operation not permitted
>>>> >
>>>>
>>>>
>>>> Thank you for these tests! would it be possible to share the brick and
>>>> client logs?
>>>>
>>>
>>> Not sure if his tests are same as my setup but here is what I end up with
>>>
>>> Volume Name: glustershard
>>> Type: Replicate
>>> Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3
>>> Status: Started
>>> Number of Bricks: 1 x 3 = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 192.168.71.10:/gluster1/shard1/1
>>> Brick2: 192.168.71.11:/gluster1/shard2/1
>>> Brick3: 192.168.71.12:/gluster1/shard3/1
>>> Options Reconfigured:
>>> features.shard-block-size: 64MB
>>> features.shard: on
>>> server.allow-insecure: on
>>> storage.owner-uid: 36
>>> storage.owner-gid: 36
>>> cluster.server-quorum-type: server
>>> cluster.quorum-type: auto
>>> network.remote-dio: enable
>>> cluster.eager-lock: enable
>>> performance.stat-prefetch: off
>>> performance.io-cache: off
>>> performance.quick-read: off
>>> cluster.self-heal-window-size: 1024
>>> cluster.background-self-heal-count: 16
>>> nfs.enable-ino32: off
>>> nfs.addr-namelookup: off
>>> nfs.disable: on
>>> performance.read-ahead: off
>>> performance.readdir-ahead: on
>>>
>>

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-26 Thread David Gossage 
Has a bug report been filed for this issue or should l I create one with
the logs and results provided so far?

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284

On Fri, Jul 22, 2016 at 12:53 PM, David Gossage  wrote:

>
>
>
> On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur  wrote:
>
>> On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen 
>> wrote:
>> > Here is a quick way how to test this:
>> > GlusterFS 3.7.13 volume with default settings with brick on ZFS
>> dataset. gluster-test1 is server and gluster-test2 is client mounting with
>> FUSE.
>> >
>> > Writing file with oflag=direct is not ok:
>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
>> count=1 bs=1024000
>> > dd: failed to open ‘file’: Invalid argument
>> >
>> > Enable network.remote-dio on Gluster Volume:
>> > [root@gluster-test1 gluster]# gluster volume set gluster
>> network.remote-dio enable
>> > volume set: success
>> >
>> > Writing small file with oflag=direct is ok:
>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
>> count=1 bs=1024000
>> > 1+0 records in
>> > 1+0 records out
>> > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s
>> >
>> > Writing bigger file with oflag=direct is ok:
>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
>> count=100 bs=1M
>> > 100+0 records in
>> > 100+0 records out
>> > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s
>> >
>> > Enable Sharding on Gluster Volume:
>> > [root@gluster-test1 gluster]# gluster volume set gluster
>> features.shard enable
>> > volume set: success
>> >
>> > Writing small file  with oflag=direct is ok:
>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
>> count=1 bs=1M
>> > 1+0 records in
>> > 1+0 records out
>> > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s
>> >
>> > Writing bigger file with oflag=direct is not ok:
>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
>> count=100 bs=1M
>> > dd: error writing ‘file3’: Operation not permitted
>> > dd: closing output file ‘file3’: Operation not permitted
>> >
>>
>>
>> Thank you for these tests! would it be possible to share the brick and
>> client logs?
>>
>
> Not sure if his tests are same as my setup but here is what I end up with
>
> Volume Name: glustershard
> Type: Replicate
> Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: 192.168.71.10:/gluster1/shard1/1
> Brick2: 192.168.71.11:/gluster1/shard2/1
> Brick3: 192.168.71.12:/gluster1/shard3/1
> Options Reconfigured:
> features.shard-block-size: 64MB
> features.shard: on
> server.allow-insecure: on
> storage.owner-uid: 36
> storage.owner-gid: 36
> cluster.server-quorum-type: server
> cluster.quorum-type: auto
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.quick-read: off
> cluster.self-heal-window-size: 1024
> cluster.background-self-heal-count: 16
> nfs.enable-ino32: off
> nfs.addr-namelookup: off
> nfs.disable: on
> performance.read-ahead: off
> performance.readdir-ahead: on
>
>
>
>  dd if=/dev/zero 
> of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/
> oflag=direct count=100 bs=1M
> 81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__
>  .trashcan/
> [root@ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/
> 192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test
> oflag=direct count=100 bs=1M
> dd: error writing 
> ‘/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test’:
> Operation not permitted
>
> creates the 64M file in expected location then the shard is 0
>
> # file: gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test
>
> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x
> trusted.bit-rot.version=0x0200579231f3000e16e7
> trusted.gfid=0xec6de302b35f427985639ca3e25d9df0
> trusted.glusterfs.shard.block-size=0x0400
>
> trusted.glusterfs.shard.file-size=0x0401
>
>
> # file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1
>
> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
> trusted.afr.dirty=0x
> trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80
>
>
>> Regards,
>> Vijay
>> ___
>> Gluster-users mailing list
>> gluster-us...@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-22 Thread David Gossage 
On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur  wrote:

> On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen 
> wrote:
> > Here is a quick way how to test this:
> > GlusterFS 3.7.13 volume with default settings with brick on ZFS dataset.
> gluster-test1 is server and gluster-test2 is client mounting with FUSE.
> >
> > Writing file with oflag=direct is not ok:
> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
> count=1 bs=1024000
> > dd: failed to open ‘file’: Invalid argument
> >
> > Enable network.remote-dio on Gluster Volume:
> > [root@gluster-test1 gluster]# gluster volume set gluster
> network.remote-dio enable
> > volume set: success
> >
> > Writing small file with oflag=direct is ok:
> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct
> count=1 bs=1024000
> > 1+0 records in
> > 1+0 records out
> > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s
> >
> > Writing bigger file with oflag=direct is ok:
> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
> count=100 bs=1M
> > 100+0 records in
> > 100+0 records out
> > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s
> >
> > Enable Sharding on Gluster Volume:
> > [root@gluster-test1 gluster]# gluster volume set gluster features.shard
> enable
> > volume set: success
> >
> > Writing small file  with oflag=direct is ok:
> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
> count=1 bs=1M
> > 1+0 records in
> > 1+0 records out
> > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s
> >
> > Writing bigger file with oflag=direct is not ok:
> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct
> count=100 bs=1M
> > dd: error writing ‘file3’: Operation not permitted
> > dd: closing output file ‘file3’: Operation not permitted
> >
>
>
> Thank you for these tests! would it be possible to share the brick and
> client logs?
>

Not sure if his tests are same as my setup but here is what I end up with

Volume Name: glustershard
Type: Replicate
Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.71.10:/gluster1/shard1/1
Brick2: 192.168.71.11:/gluster1/shard2/1
Brick3: 192.168.71.12:/gluster1/shard3/1
Options Reconfigured:
features.shard-block-size: 64MB
features.shard: on
server.allow-insecure: on
storage.owner-uid: 36
storage.owner-gid: 36
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.quick-read: off
cluster.self-heal-window-size: 1024
cluster.background-self-heal-count: 16
nfs.enable-ino32: off
nfs.addr-namelookup: off
nfs.disable: on
performance.read-ahead: off
performance.readdir-ahead: on



 dd if=/dev/zero
of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/
oflag=direct count=100 bs=1M
81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__
 .trashcan/
[root@ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/
192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test
oflag=direct count=100 bs=1M
dd: error writing
‘/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test’:
Operation not permitted

creates the 64M file in expected location then the shard is 0

# file: gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.bit-rot.version=0x0200579231f3000e16e7
trusted.gfid=0xec6de302b35f427985639ca3e25d9df0
trusted.glusterfs.shard.block-size=0x0400
trusted.glusterfs.shard.file-size=0x0401


# file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x
trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80


> Regards,
> Vijay
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
[2016-07-22 16:26:44.645166] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.13 (args: /usr/sbin/glusterfs --volfile-server=192.168.71.11 --volfile-server=192.168.71.10 --volfile-server=192.168.71.12 --volfile-id=/glustershard /rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard)
[2016-07-22 16:26:44.655674] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-07-22 16:26:44.661967] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-07-22 16:26:44.662718] I [MSGID: 114020] [client.c:2106:notify] 0-glustershard-client-0:

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-22 Thread David Gossage 
On Fri, Jul 22, 2016 at 8:12 AM, Vijay Bellur  wrote:

> 2016-07-22 1:54 GMT-04:00 Frank Rothenstein <
> f.rothenst...@bodden-kliniken.de>:
> > The point is that even if all other backend storage filesystems do
> correctly
> > untill 3.7.11 there was no error on ZFS. Something happened nobody ever
> > could explain in the release of 3.7.12 that makes FUSE-mount _in ovirt_
> (it
> > partly uses dd with iflag=direct  , using iflag=direct yourself gives
> also
> > errors on the FUSE-mounts ) unusable.
> >
> > So 3.7.11 is the last usable version when using ZFS on bricks, afaik.
> >
>
> Can you please share the exact dd command that causes this problem?
>

I want to say it was this one though my logs for vdsm going back that far
have rolled off

 /usr/bin/dd
if=/rhev/data-center/mnt/glusterSD/ccgl1.gl.local:GLUSTER1/7c73a8dd-a72e-4556-ac88-7f6813131e64/dom_md/metadata
iflag=direct of=/dev/null bs=4096 count=1

>
> Thanks,
> Vijay
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-22 Thread David Gossage 
On Fri, Jul 22, 2016 at 8:23 AM, Samuli Heinonen 
wrote:

>
> > On 21 Jul 2016, at 20:48, David Gossage 
> wrote:
> >
> > Wonder if this may be related at all
> >
> > * #1347553: O_DIRECT support for sharding
> > https://bugzilla.redhat.com/show_bug.cgi?id=1347553
> >
> > Is it possible to downgrade from 3.8 back to 3.7.x
> >
> > Building test box right now anyway but wondering.
> >
>
> Have you been able to do any testing yet?
>

Had time to get box built up to point of getting zfs pool made before left
work yesterday.  hope to have working gluster volumes on various fs
backends by end of day.

Figure 1 volume on xfs with sharding, 1 volume on zfs sharded, 1 on zfs no
shards, and maybe ill make 1 zvol with xfs on top of it also to see what
happens


> "O_DIRECT support for sharding" has been also included in 3.7.12. Is this
> problem occurring only when sharding is enabled? Is it possible that it
> requires direct I/O all the way to the bricks with sharding even when
> network.remote-dio is enabled?
>

It's possible though at time of update when I had issues getting it to stay
connected to ovirt I had just turned sharding on and as of yet had no
sharded images.  I also turned feature off during tests with no benefit in
stability.

>
> Best regards,
> Samuli Heinonen
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-22 Thread David Gossage 
2016-07-22 2:32 GMT-05:00 Frank Rothenstein <
f.rothenst...@bodden-kliniken.de>:

> I can't tell myself, I'm using the ovirt-4.0-centos-gluster37 repo
> (from ovirt-release40). I have a second gluster-cluster as storage, I
> didn't dare to upgrade, as it simply works...not as an ovirt/vm storage.
>
> Am Freitag, den 22.07.2016, 08:28 +0200 schrieb Gandalf Corvotempesta:
>
> Il 22 lug 2016 07:54, "Frank Rothenstein" <
> f.rothenst...@bodden-kliniken.de> ha scritto:
> >
> > So 3.7.11 is the last usable version when using ZFS on bricks, afaik.
>
> Even with 3.8 this issue is present?
>
>
I believe Lindsay may have tested 3.8 and found is their still as well.



>
>
> --
>
>
>
>
> __
> BODDEN-KLINIKEN Ribnitz-Damgarten GmbH
> Sandhufe 2
> 18311 Ribnitz-Damgarten
>
> Telefon: 03821-700-0
> Fax:   03821-700-240
>
> E-Mail: i...@bodden-kliniken.de   Internet: http://www.bodden-kliniken.de
>
>
> Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-Nr.: 
> 079/133/40188
>
> Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. Falko Milski
>
>
> Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten Adressaten 
> bestimmt. Wenn Sie nicht der vorge-
>
> sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, beachten 
> Sie bitte, dass jede Form der Veröf-
>
> fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-Mail 
> unzulässig ist. Wir bitten Sie, sofort den
> Absender zu informieren und die E-Mail zu löschen.
>
>
>  Bodden-Kliniken Ribnitz-Damgarten GmbH 2016
> *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-21 Thread David Gossage 
On Thu, Jul 21, 2016 at 2:48 PM, Kaleb KEITHLEY  wrote:

> On 07/21/2016 02:38 PM, Samuli Heinonen wrote:
> > Hi all,
> >
> > I’m running oVirt 3.6 and Gluster 3.7 with ZFS backend.
> > ...
> > Afaik ZFS on Linux doesn’t support aio. Has there been some changes to
> GlusterFS regarding aio?
> >
> Was under impression it did
>

https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.6.4


   - New asynchronous I/O (AIO) support.



> Boy, if that isn't a smoking gun, I don't know what is.
>
> --
>
> Kaleb
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-21 Thread David Gossage 
On Thu, Jul 21, 2016 at 12:48 PM, David Gossage  wrote:

> On Thu, Jul 21, 2016 at 9:58 AM, David Gossage <
> dgoss...@carouselchecks.com> wrote:
>
>> On Thu, Jul 21, 2016 at 9:52 AM, Niels de Vos  wrote:
>>
>>> On Sun, Jul 10, 2016 at 10:49:52AM +1000, Lindsay Mathieson wrote:
>>> > Did a quick test this morning - 3.7.13 is now working with libgfapi -
>>> yay!
>>> >
>>> >
>>> > However I do have to enable write-back or write-through caching in qemu
>>> > before the vm's will start, I believe this is to do with aio support.
>>> Not a
>>> > problem for me.
>>> >
>>> > I see there are settings for storage.linux-aio and storage.bd-aio -
>>> not sure
>>> > as to whether they are relevant or which ones to play with.
>>>
>>> Both storage.*-aio options are used by the brick processes. Depending on
>>> what type of brick you have (linux = filesystem, bd = LVM Volume Group)
>>> you could enable the one or the other.
>>>
>>> We do have a strong suggestion to set these "gluster volume group .."
>>> options:
>>>
>>> https://github.com/gluster/glusterfs/blob/master/extras/group-virt.example
>>>
>>> From those options, network.remote-dio seems most related to your aio
>>> theory. It was introduced with http://review.gluster.org/4460 that
>>> contains some more details.
>>>
>>
>
> Wonder if this may be related at all
>
> * #1347553: O_DIRECT support for sharding
> https://bugzilla.redhat.com/show_bug.cgi?id=1347553
>
> Is it possible to downgrade from 3.8 back to 3.7.x
>
> Building test box right now anyway but wondering.
>

May be anecdotal with small sample size but the few people who have had
issue all seemed to have zfs backed gluster volumes.

Now that i recall back to the day I updated.  The gluster volume on xfs I
use for my hosted engine never had issues.


>
>
>
>>
>> Thanks with the exception of stat-prefetch I have those enabled
>> I could try turning that back off though at the time of update to 3.7.13
>> it was off.  I didnt turn it back on till later in next week after
>> downgrading back to 3.7.11.
>>
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
>> Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
>> Brick3: ccgl4.gl.local:/gluster1/BRICK1/1
>> Options Reconfigured:
>> diagnostics.brick-log-level: WARNING
>> features.shard-block-size: 64MB
>> features.shard: on
>> performance.readdir-ahead: on
>> storage.owner-uid: 36
>> storage.owner-gid: 36
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.stat-prefetch: on
>> cluster.eager-lock: enable
>> network.remote-dio: enable
>> cluster.quorum-type: auto
>> cluster.server-quorum-type: server
>> server.allow-insecure: on
>> cluster.self-heal-window-size: 1024
>> cluster.background-self-heal-count: 16
>> performance.strict-write-ordering: off
>> nfs.disable: on
>> nfs.addr-namelookup: off
>> nfs.enable-ino32: off
>>
>>
>>> HTH,
>>> Niels
>>>
>>> ___
>>> Gluster-users mailing list
>>> gluster-us...@gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-21 Thread David Gossage 
On Thu, Jul 21, 2016 at 9:58 AM, David Gossage 
wrote:

> On Thu, Jul 21, 2016 at 9:52 AM, Niels de Vos  wrote:
>
>> On Sun, Jul 10, 2016 at 10:49:52AM +1000, Lindsay Mathieson wrote:
>> > Did a quick test this morning - 3.7.13 is now working with libgfapi -
>> yay!
>> >
>> >
>> > However I do have to enable write-back or write-through caching in qemu
>> > before the vm's will start, I believe this is to do with aio support.
>> Not a
>> > problem for me.
>> >
>> > I see there are settings for storage.linux-aio and storage.bd-aio - not
>> sure
>> > as to whether they are relevant or which ones to play with.
>>
>> Both storage.*-aio options are used by the brick processes. Depending on
>> what type of brick you have (linux = filesystem, bd = LVM Volume Group)
>> you could enable the one or the other.
>>
>> We do have a strong suggestion to set these "gluster volume group .."
>> options:
>>
>> https://github.com/gluster/glusterfs/blob/master/extras/group-virt.example
>>
>> From those options, network.remote-dio seems most related to your aio
>> theory. It was introduced with http://review.gluster.org/4460 that
>> contains some more details.
>>
>

Wonder if this may be related at all

* #1347553: O_DIRECT support for sharding
https://bugzilla.redhat.com/show_bug.cgi?id=1347553

Is it possible to downgrade from 3.8 back to 3.7.x

Building test box right now anyway but wondering.



>
> Thanks with the exception of stat-prefetch I have those enabled
> I could try turning that back off though at the time of update to 3.7.13
> it was off.  I didnt turn it back on till later in next week after
> downgrading back to 3.7.11.
>
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
> Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
> Brick3: ccgl4.gl.local:/gluster1/BRICK1/1
> Options Reconfigured:
> diagnostics.brick-log-level: WARNING
> features.shard-block-size: 64MB
> features.shard: on
> performance.readdir-ahead: on
> storage.owner-uid: 36
> storage.owner-gid: 36
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: on
> cluster.eager-lock: enable
> network.remote-dio: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> server.allow-insecure: on
> cluster.self-heal-window-size: 1024
> cluster.background-self-heal-count: 16
> performance.strict-write-ordering: off
> nfs.disable: on
> nfs.addr-namelookup: off
> nfs.enable-ino32: off
>
>
>> HTH,
>> Niels
>>
>> ___
>> Gluster-users mailing list
>> gluster-us...@gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-21 Thread David Gossage 
On Thu, Jul 21, 2016 at 9:52 AM, Niels de Vos  wrote:

> On Sun, Jul 10, 2016 at 10:49:52AM +1000, Lindsay Mathieson wrote:
> > Did a quick test this morning - 3.7.13 is now working with libgfapi -
> yay!
> >
> >
> > However I do have to enable write-back or write-through caching in qemu
> > before the vm's will start, I believe this is to do with aio support.
> Not a
> > problem for me.
> >
> > I see there are settings for storage.linux-aio and storage.bd-aio - not
> sure
> > as to whether they are relevant or which ones to play with.
>
> Both storage.*-aio options are used by the brick processes. Depending on
> what type of brick you have (linux = filesystem, bd = LVM Volume Group)
> you could enable the one or the other.
>
> We do have a strong suggestion to set these "gluster volume group .."
> options:
>
> https://github.com/gluster/glusterfs/blob/master/extras/group-virt.example
>
> From those options, network.remote-dio seems most related to your aio
> theory. It was introduced with http://review.gluster.org/4460 that
> contains some more details.
>

Thanks with the exception of stat-prefetch I have those enabled
I could try turning that back off though at the time of update to 3.7.13 it
was off.  I didnt turn it back on till later in next week after downgrading
back to 3.7.11.

Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
Brick3: ccgl4.gl.local:/gluster1/BRICK1/1
Options Reconfigured:
diagnostics.brick-log-level: WARNING
features.shard-block-size: 64MB
features.shard: on
performance.readdir-ahead: on
storage.owner-uid: 36
storage.owner-gid: 36
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: on
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
server.allow-insecure: on
cluster.self-heal-window-size: 1024
cluster.background-self-heal-count: 16
performance.strict-write-ordering: off
nfs.disable: on
nfs.addr-namelookup: off
nfs.enable-ino32: off


> HTH,
> Niels
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-21 Thread David Gossage 
On Thu, Jul 21, 2016 at 9:33 AM, Kaleb KEITHLEY  wrote:

> On 07/21/2016 10:19 AM, David Gossage wrote:
> > Has their been any release notes or bug reports about the removal of aio
> > support being intentional?
>
> Build logs of 3.7.13 on Fedora and Ubuntu PPA (Launchpad) show that when
> `configure` ran during the build it reported that Linux AIO was enabled.
>
> What packages are you using? On which Linux distribution?
>

 glusterfs-epel.repo
http://download.gluster.org/pub/gluster/glusterfs/3.7/
glusterfs-client-xlators-3.7.11-1.el7.x86_64
glusterfs-cli-3.7.11-1.el7.x86_64
glusterfs-libs-3.7.11-1.el7.x86_64
glusterfs-3.7.11-1.el7.x86_64
glusterfs-fuse-3.7.11-1.el7.x86_64
glusterfs-server-3.7.11-1.el7.x86_64
glusterfs-api-3.7.11-1.el7.x86_64


Had to downgrade from 3.7.12/13 as couldn't keep storage connection stable.



Maybe issue is different then.  I just know updating to 3.7.12 and 13 a few
of us in mail list of ovirt and gluster have been having issues that only
resolve by changing how images are accessed over fuse, or moving to NFS
away from fuse all together.

I'll see if I can gather enough useful information to file a bug report.


> You might like to file a bug report at
> https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
>
>
> --
>
> Kaleb
>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] 3.7.13 & proxmox/qemu

2016-07-21 Thread David Gossage 
Has their been any release notes or bug reports about the removal of aio
support being intentional?  In the case of proxmox it seems to be an easy
workaround to resolve more or less.

However In the case of oVirt I can change cache method per VM with a custom
property key, but the dd process that tests storage backends has in the
python scripts the direct flag hard coded in from what I have found so far.

I could potentially swap to nfs-ganesha but again in ovirt exporting and
importing a storage domain with a differing protocol is not necessarily
what you want to be doing if you can avoid it.  I'd probably end up
creating a 2nd gluster volume and have to migrate disk by disk.

Just trying to figure out what the roadmap of this is and what resolution I
should be ultimately heading for.

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284

On Sat, Jul 9, 2016 at 7:49 PM, Lindsay Mathieson <
lindsay.mathie...@gmail.com> wrote:

> Did a quick test this morning - 3.7.13 is now working with libgfapi - yay!
>
>
> However I do have to enable write-back or write-through caching in qemu
> before the vm's will start, I believe this is to do with aio support. Not a
> problem for me.
>
> I see there are settings for storage.linux-aio and storage.bd-aio - not
> sure as to whether they are relevant or which ones to play with.
>
> thanks,
>
> --
> Lindsay Mathieson
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel

Re: [Gluster-devel] [Gluster-users] Fwd: dht_is_subvol_filled messages on client

2016-05-05 Thread David Gossage 
On Thu, May 5, 2016 at 3:28 AM, Serkan Çoban  wrote:

> Hi,
>
> You can find the output below link:
> https://www.dropbox.com/s/wzrh5yp494ogksc/status_detail.txt?dl=0
>
> Thanks,
> Serkan
>

Maybe not issue, but playing one of these things is not like the other I
notice of all the bricks only one seems to be different at a quick glance

Brick: Brick 1.1.1.235:/bricks/20
TCP Port : 49170
RDMA Port: 0
Online   : Y
Pid  : 26736
File System  : ext4
Device   : /dev/mapper/vol0-vol_root
Mount Options: rw,relatime,data=ordered
Inode Size   : 256
Disk Space Free  : 86.1GB
Total Disk Space : 96.0GB
Inode Count  : 6406144
Free Inodes  : 6381374

Every other brick seems to be 7TB and xfs but this one.




>
> On Thu, May 5, 2016 at 9:33 AM, Xavier Hernandez 
> wrote:
> > Can you post the result of 'gluster volume status v0 detail' ?
> >
> >
> > On 05/05/16 06:49, Serkan Çoban wrote:
> >>
> >> Hi, Can anyone suggest something for this issue? df, du has no issue
> >> for the bricks yet one subvolume not being used by gluster..
> >>
> >> On Wed, May 4, 2016 at 4:40 PM, Serkan Çoban 
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I changed cluster.min-free-inodes to "0". Remount the volume on
> >>> clients. inode full messages not coming to syslog anymore but I see
> >>> disperse-56 subvolume still not being used.
> >>> Anything I can do to resolve this issue? Maybe I can destroy and
> >>> recreate the volume but I am not sure It will fix this issue...
> >>> Maybe the disperse size 16+4 is too big should I change it to 8+2?
> >>>
> >>> On Tue, May 3, 2016 at 2:36 PM, Serkan Çoban 
> >>> wrote:
> 
>  I also checked the df output all 20 bricks are same like below:
>  /dev/sdu1 7.3T 34M 7.3T 1% /bricks/20
> 
>  On Tue, May 3, 2016 at 1:40 PM, Raghavendra G <
> raghaven...@gluster.com>
>  wrote:
> >
> >
> >
> > On Mon, May 2, 2016 at 11:41 AM, Serkan Çoban  >
> > wrote:
> >>
> >>
> >>> 1. What is the out put of du -hs ? Please get this
> >>> information for each of the brick that are part of disperse.
> >
> >
> >
> > Sorry. I needed df output of the filesystem containing brick. Not du.
> > Sorry
> > about that.
> >
> >>
> >> There are 20 bricks in disperse-56 and the du -hs output is like:
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 1.8M /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >> 80K /bricks/20
> >>
> >> I see that gluster is not writing to this disperse set. All other
> >> disperse sets are filled 13GB but this one is empty. I see directory
> >> structure created but no files in directories.
> >> How can I fix the issue? I will try to rebalance but I don't think
> it
> >> will write to this disperse set...
> >>
> >>
> >>
> >> On Sat, Apr 30, 2016 at 9:22 AM, Raghavendra G
> >> 
> >> wrote:
> >>>
> >>>
> >>>
> >>> On Fri, Apr 29, 2016 at 12:32 AM, Serkan Çoban
> >>> 
> >>> wrote:
> 
> 
>  Hi, I cannot get an answer from user list, so asking to devel
> list.
> 
>  I am getting [dht-diskusage.c:277:dht_is_subvol_filled] 0-v0-dht:
>  inodes on subvolume 'v0-disperse-56' are at (100.00 %), consider
>  adding more bricks.
> 
>  message on client logs.My cluster is empty there are only a couple
>  of
>  GB files for testing. Why this message appear in syslog?
> >>>
> >>>
> >>>
> >>> dht uses disk usage information from backend export.
> >>>
> >>> 1. What is the out put of du -hs ? Please get this
> >>> information for each of the brick that are part of disperse.
> >>> 2. Once you get du information from each brick, the value seen by
> dht
> >>> will
> >>> be based on how cluster/disperse aggregates du info (basically
> statfs
> >>> fop).
> >>>
> >>> The reason for 100% disk usage may be,
> >>> In case of 1, backend fs might be shared by data other than brick.
> >>> In case of 2, some issues with aggregation.
> >>>
>  Is is safe to
>  ignore it?
> >>>
> >>>
> >>>
> >>> dht will try not to have data files on the subvol in question
> >>> (v0-disperse-56). Hence lookup cost will be two hops for files
> >>> hashing
> >>> to
> >>> disperse-56 (note that other fops like read/write/open still have
> the
> >>> cost
> >>> of single hop and dont suffer from this penalty). Other than that

Re: [Gluster-devel] [Gluster-users] AFR arbiter volumes

2015-09-11 Thread David Gossage 
Once the volume is created as an Arbiter volume can it at a later time be
changed to a replica 3 with all bricks containing data?

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284

On Tue, Sep 8, 2015 at 8:46 PM, Ravishankar N 
wrote:

> Sending out this mail for awareness/ feedback.
>
> -
> *What:*
> Since glusterfs-3.7,  AFR supports creation of arbiter volumes. These are
> a special type of replica 3 gluster volume where the 3rd brick  is (always)
> configured as an arbiter node.What this means is that the 3rd brick will
> store only the file name and metadata (including gluster xattrs), but does
> not contain any data. Arbiter volumes prevent split-brains and consumes
> lesser space than a normal replica 3 volume and provides better consistency
> and availability than a replica 2 volume.
>
> *How:*
> You can create an arbiter volume with the following command:
>
> * gluster volume create  replica 3 arbiter 1 host1:brick1
> host2:brick2 host3:brick3*
>
> Note that the syntax is similar to creating a normal replica 3 volume with
> the exception of the *arbiter 1* keyword. As seen in the command above,
> the only permissible values for the replica count and arbiter count are 3
> and 1 respectively. Also, the 3rd brick is always chosen as the arbiter
> brick and it is currently not configurable to have any other brick as the
> arbiter.
>
> *Client/ Mount behaviour:*
> By default, client quorum (cluster.quorum-type) is set to auto for a
> replica 3 volume (including arbiter volumes) when it is created; i.e. at
> least 2 bricks need to be up to satisfy quorum and to allow writes. This
> setting is not to be changed for arbiter volumes also. Additionally, the
> arbiter volume has some additional checks to prevent files from ending up
> in split-brain:
>
> * Clients take full file locks when writing to a file as opposed to
> range locks in a normal replica 3 volume.
>
> * If 2 bricks are up and if one of them is the arbiter (i.e. the 3rd
> brick) and it blames the other up brick, then all FOPS will fail with
> ENOTCONN (Transport endpoint is not connected). If the arbiter doesn't
> blame the other brick, FOPS will be allowed to proceed. 'Blaming' here is
> w.r.t the values of AFR changelog extended attributes.
>
> * If 2 bricks are up and the arbiter is down, then FOPS will be
> allowed.
>
> * In all cases, if there is only one source before the FOP is
> initiated and if the FOP fails on that source, the application will receive
> ENOTCONN.
>
> Note: It is possible to see if a replica 3 volume has arbiter
> configuration from the mount point. If*
> $mount_point/.meta/graphs/active/$V0-replicate-0/options/arbiter-count*
> exists and its value is 1, then it is an arbiter volume. Also the client
> volume graph will have arbiter-count as a xlator option for AFR translators.
>
> *Self-heal daemon behaviour:*
>
> Since the arbiter brick does not store any data for the files, it cannot
> be used as a source for data self-heal. For example if there are 2 source
> bricks B2 and B3 (B3 being arbiter brick) and B2 is down, then
> data-self-heal will not happen from B3 to sink brick B1, and will be
> pending until B2 comes up and heal can happen from it. Note that metadata
> and entry self-heals can still happen from B3 if it is one of the sources.
>
>
> -
>
> Please provide feedback if you have tried it out.
> *If you ever encounter a split-brain while using the arbiter volume, it is
> a BUG - do report!*
> We have had users asking for a way to convert existing replica 2 volumes
> to arbiter volumes- this is definitely in our to-do list, in addition to
> some performance optimizations.
>
> Thanks,
> Ravi
>
> ___
> Gluster-users mailing list
> gluster-us...@gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel