So far both dd commands that failed previously worked fine on 3.7.14 Once I deleted old content from test volume it mounted to oVirt via storage add when previously it would error out. I am now creating a test VM with default disk caching settings (pretty sure oVirt is defaulting to none rather than writeback/through). So far all shards are being created properly.
Load is sky rocketing but I have all 3 gluster bricks running off 1 hard drive on test box so I would expect horrible io/load issues with that. Very promising so far. Thank you developers for your help in working through this. Once I have the VM installed and running will test for a few days and make sure it doesn't have any freeze or locking issues then will roll this out to working cluster. *David Gossage* *Carousel Checks Inc. | System Administrator* *Office* 708.613.2284 On Wed, Jul 27, 2016 at 8:37 AM, David Gossage <dgoss...@carouselchecks.com> wrote: > On Tue, Jul 26, 2016 at 9:38 PM, Krutika Dhananjay <kdhan...@redhat.com> > wrote: > >> Yes please, could you file a bug against glusterfs for this issue? >> > > https://bugzilla.redhat.com/show_bug.cgi?id=1360785 > > >> >> >> -Krutika >> >> On Wed, Jul 27, 2016 at 1:39 AM, David Gossage < >> dgoss...@carouselchecks.com> wrote: >> >>> Has a bug report been filed for this issue or should l I create one with >>> the logs and results provided so far? >>> >>> *David Gossage* >>> *Carousel Checks Inc. | System Administrator* >>> *Office* 708.613.2284 >>> >>> On Fri, Jul 22, 2016 at 12:53 PM, David Gossage < >>> dgoss...@carouselchecks.com> wrote: >>> >>>> >>>> >>>> >>>> On Fri, Jul 22, 2016 at 9:37 AM, Vijay Bellur <vbel...@redhat.com> >>>> wrote: >>>> >>>>> On Fri, Jul 22, 2016 at 10:03 AM, Samuli Heinonen < >>>>> samp...@neutraali.net> wrote: >>>>> > Here is a quick way how to test this: >>>>> > GlusterFS 3.7.13 volume with default settings with brick on ZFS >>>>> dataset. gluster-test1 is server and gluster-test2 is client mounting with >>>>> FUSE. >>>>> > >>>>> > Writing file with oflag=direct is not ok: >>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>>>> count=1 bs=1024000 >>>>> > dd: failed to open ‘file’: Invalid argument >>>>> > >>>>> > Enable network.remote-dio on Gluster Volume: >>>>> > [root@gluster-test1 gluster]# gluster volume set gluster >>>>> network.remote-dio enable >>>>> > volume set: success >>>>> > >>>>> > Writing small file with oflag=direct is ok: >>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct >>>>> count=1 bs=1024000 >>>>> > 1+0 records in >>>>> > 1+0 records out >>>>> > 1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s >>>>> > >>>>> > Writing bigger file with oflag=direct is ok: >>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>>>> count=100 bs=1M >>>>> > 100+0 records in >>>>> > 100+0 records out >>>>> > 104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s >>>>> > >>>>> > Enable Sharding on Gluster Volume: >>>>> > [root@gluster-test1 gluster]# gluster volume set gluster >>>>> features.shard enable >>>>> > volume set: success >>>>> > >>>>> > Writing small file with oflag=direct is ok: >>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>>>> count=1 bs=1M >>>>> > 1+0 records in >>>>> > 1+0 records out >>>>> > 1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s >>>>> > >>>>> > Writing bigger file with oflag=direct is not ok: >>>>> > [root@gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct >>>>> count=100 bs=1M >>>>> > dd: error writing ‘file3’: Operation not permitted >>>>> > dd: closing output file ‘file3’: Operation not permitted >>>>> > >>>>> >>>>> >>>>> Thank you for these tests! would it be possible to share the brick and >>>>> client logs? >>>>> >>>> >>>> Not sure if his tests are same as my setup but here is what I end up >>>> with >>>> >>>> Volume Name: glustershard >>>> Type: Replicate >>>> Volume ID: 0cc4efb6-3836-4caa-b24a-b3afb6e407c3 >>>> Status: Started >>>> Number of Bricks: 1 x 3 = 3 >>>> Transport-type: tcp >>>> Bricks: >>>> Brick1: 192.168.71.10:/gluster1/shard1/1 >>>> Brick2: 192.168.71.11:/gluster1/shard2/1 >>>> Brick3: 192.168.71.12:/gluster1/shard3/1 >>>> Options Reconfigured: >>>> features.shard-block-size: 64MB >>>> features.shard: on >>>> server.allow-insecure: on >>>> storage.owner-uid: 36 >>>> storage.owner-gid: 36 >>>> cluster.server-quorum-type: server >>>> cluster.quorum-type: auto >>>> network.remote-dio: enable >>>> cluster.eager-lock: enable >>>> performance.stat-prefetch: off >>>> performance.io-cache: off >>>> performance.quick-read: off >>>> cluster.self-heal-window-size: 1024 >>>> cluster.background-self-heal-count: 16 >>>> nfs.enable-ino32: off >>>> nfs.addr-namelookup: off >>>> nfs.disable: on >>>> performance.read-ahead: off >>>> performance.readdir-ahead: on >>>> >>>> >>>> >>>> dd if=/dev/zero >>>> of=/rhev/data-center/mnt/glusterSD/192.168.71.11\:_glustershard/ >>>> oflag=direct count=100 bs=1M >>>> 81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/ __DIRECT_IO_TEST__ >>>> .trashcan/ >>>> [root@ccengine2 ~]# dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ >>>> 192.168.71.11\:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >>>> oflag=direct count=100 bs=1M >>>> dd: error writing >>>> ‘/rhev/data-center/mnt/glusterSD/192.168.71.11:_glustershard/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test’: >>>> Operation not permitted >>>> >>>> creates the 64M file in expected location then the shard is 0 >>>> >>>> # file: >>>> gluster1/shard1/1/81e19cd3-ae45-449c-b716-ec3e4ad4c2f0/images/test >>>> >>>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>> trusted.afr.dirty=0x000000000000000000000000 >>>> trusted.bit-rot.version=0x0200000000000000579231f3000e16e7 >>>> trusted.gfid=0xec6de302b35f427985639ca3e25d9df0 >>>> trusted.glusterfs.shard.block-size=0x0000000004000000 >>>> >>>> trusted.glusterfs.shard.file-size=0x0000000004000000000000000000000000000000000000010000000000000000 >>>> >>>> >>>> # file: gluster1/shard1/1/.shard/ec6de302-b35f-4279-8563-9ca3e25d9df0.1 >>>> >>>> security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000 >>>> trusted.afr.dirty=0x000000000000000000000000 >>>> trusted.gfid=0x2bfd3cc8a727489b9a0474241548fe80 >>>> >>>> >>>>> Regards, >>>>> Vijay >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users@gluster.org >>>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>> >>>> >>>> >>> >>> _______________________________________________ >>> Gluster-devel mailing list >>> gluster-de...@gluster.org >>> http://www.gluster.org/mailman/listinfo/gluster-devel >>> >> >> >
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users