[Gluster-users] trusted.ec.dirty attribute

[Dmitri Chebotarov]

Hi

I've got a problem on a EC volume where heal doesn't seem to work for just
few files (heal info shows no progress for few days, Warning on the client
mount)

I ran

getfattr -m . -d -e hex /

across all servers in the cluster and 'trusted.ec.dirty' attr is non-zero
on all files which don't heal.

Is it normal? Any way to correct it?
It's my understanding it should
be trusted.ec.dirty=0x for 'good' files.

Out of 12 copies of the files, 11 have the same non-zero value
(0x00111592), and 12th file has a different
non-zero value.

Thank you.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [ovirt-users] slow performance with export storage on glusterfs

[Dmitri Chebotarov]

Hello

If you use Gluster as FUSE mount it's always slower than you expect it to
be.
If you want to get better performance out of your oVirt/Gluster storage,
try the following:

- create a Linux VM in your oVirt environment, assign 4/8/12 virtual disks
(Virtual disks are located on your Gluster storage volume).
- Boot/configure the VM, then use LVM to create VG/LV with 4 stripes
(lvcreate -i 4) and use all 4/8/12 virtual disks as PVS.
- then install NFS server and export LV you created in previous step, use
the NFS export as export domain in oVirt/RHEV.

You should get wire speed when you use multiple stripes on Gluster storage,
FUSE mount on oVirt host will fan out requests to all 4 servers.
Gluster is very good at distributed/parallel workloads, but when you use
direct Gluster FUSE mount for Export domain you only have one data stream,
which is fragmented even more my multiple writes/reads that Gluster needs
to do to save your data on all member servers.



On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis  wrote:

> What about mounting over nfs instead of the fuse client. Or maybe
> libgfapi. Is that available for export domains
>
> On Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka  wrote:
>
>> On 11/24/2017 06:41 AM, Sahina Bose wrote:
>> >
>> >
>> > On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka > > > wrote:
>> >
>> > Hi,
>> >
>> > On 11/22/2017 07:30 PM, Nir Soffer wrote:
>> > > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka > 
>> > > >> wrote:
>> > >
>> > > Hi,
>> > >
>> > > I am trying realize why is exporting of vm to export storage
>> on
>> > > glusterfs such slow.
>> > >
>> > > I am using oVirt and RHV, both instalations on version 4.1.7.
>> > >
>> > > Hosts have dedicated nics for rhevm network - 1gbps, data
>> > storage itself
>> > > is on FC.
>> > >
>> > > GlusterFS cluster lives separate on 4 dedicated hosts. It has
>> > slow disks
>> > > but I can achieve about 200-400mbit throughput in other
>> > applications (we
>> > > are using it for "cold" data, backups mostly).
>> > >
>> > > I am using this glusterfs cluster as backend for export
>> > storage. When I
>> > > am exporting vm I can see only about 60-80mbit throughput.
>> > >
>> > > What could be the bottleneck here?
>> > >
>> > > Could it be qemu-img utility?
>> > >
>> > > vdsm  97739  0.3  0.0 354212 29148 ?S>  0:06
>> > > /usr/bin/qemu-img convert -p -t none -T none -f raw
>> > >
>> >  /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
>> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
>> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> > > -O raw
>> > >
>> >  /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
>> export/81094499-a392-4ea2-b081-7c6288fbb636/images/
>> ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> > >
>> > > Any idea how to make it work faster or what throughput should
>> I
>> > > expected?
>> > >
>> > >
>> > > gluster storage operations are using fuse mount - so every write:
>> > > - travel to the kernel
>> > > - travel back to the gluster fuse helper process
>> > > - travel to all 3 replicas - replication is done on client side
>> > > - return to kernel when all writes succeeded
>> > > - return to caller
>> > >
>> > > So gluster will never set any speed record.
>> > >
>> > > Additionally, you are copying from raw lv on FC - qemu-img cannot
>> do
>> > > anything
>> > > smart and avoid copying unused clusters. Instead if copies
>> > gigabytes of
>> > > zeros
>> > > from FC.
>> >
>> > ok, it does make sense
>> >
>> > > However 7.5-10 MiB/s sounds too slow.
>> > >
>> > > I would try to test with dd - how much time it takes to copy
>> > > the same image from FC to your gluster storage?
>> > >
>> > > dd
>> > > if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/
>> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc-
>> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e
>> > > of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__
>> export/81094499-a392-4ea2-b081-7c6288fbb636/__test__
>> > > bs=8M oflag=direct status=progress
>> >
>> > unfrotunately dd performs the same
>> >
>> > 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s
>> >
>> >
>> > > If dd can do this faster, please ask on qemu-discuss mailing list:
>> > > https://lists.nongnu.org/mailman/listinfo/qemu-discuss
>> > 
>> > >
>> > > If both give similar results, I think asking in gluster mailing
>> list
>> > > about this can help. Maybe your gluster setup c

Re: [Gluster-users] warning spam in the logs after tiering experiment

[Dmitri Chebotarov]

Hi Alastair

I apologize my email is off topic, but I also have strange issues on a
volume and it happened to be volume with hot-tier attached to it in past.

Can you do me a favor and test something on the volume where you used to
have hot-tier?

I'm working with RH on issue where glusterfs fuse client crashes while
doing 'grep' on files.

Here are the steps:

On the volume where you had hot-tier enabled:

# generate dataset for testing:

mkdir uuid-set; cd uuid-set
for i in $(seq 40); do uuidgen | awk {'print "mkdir "$1"; echo test >>
"$1"/"$1".meta"'}; done | sh

# run grep

grep test */*.meta

#Running 'grep' may crash glusterfs-client. If it doesn't crash, do the
following:

rsync -varP uuid-set/ uuid-set2
grep test uuid-set2/*/*.meta

Thank you.

On Wed, Oct 18, 2017 at 1:27 PM, Alastair Neil 
wrote:

> forgot to mention Gluster version 3.10.6
>
> On 18 October 2017 at 13:26, Alastair Neil  wrote:
>
>> a short while ago I experimented with tiering on one of my volumes.  I
>> decided it was not working out so I removed the tier.  I now have spam in
>> the glusterd.log evert 7 seconds:
>>
>> [2017-10-18 17:17:29.578327] W [socket.c:3207:socket_connect] 0-tierd:
>> Ignore failed connection attempt on 
>> /var/run/gluster/2e3df1c501d0a19e5076304179d1e43e.socket,
>> (No such file or directory)
>> [2017-10-18 17:17:36.579276] W [socket.c:3207:socket_connect] 0-tierd:
>> Ignore failed connection attempt on 
>> /var/run/gluster/2e3df1c501d0a19e5076304179d1e43e.socket,
>> (No such file or directory)
>> [2017-10-18 17:17:43.580238] W [socket.c:3207:socket_connect] 0-tierd:
>> Ignore failed connection attempt on 
>> /var/run/gluster/2e3df1c501d0a19e5076304179d1e43e.socket,
>> (No such file or directory)
>> [2017-10-18 17:17:50.581185] W [socket.c:3207:socket_connect] 0-tierd:
>> Ignore failed connection attempt on 
>> /var/run/gluster/2e3df1c501d0a19e5076304179d1e43e.socket,
>> (No such file or directory)
>> [2017-10-18 17:17:57.582136] W [socket.c:3207:socket_connect] 0-tierd:
>> Ignore failed connection attempt on 
>> /var/run/gluster/2e3df1c501d0a19e5076304179d1e43e.socket,
>> (No such file or directory)
>> [2017-10-18 17:18:04.583148] W [socket.c:3207:socket_connect] 0-tierd:
>> Ignore failed connection attempt on 
>> /var/run/gluster/2e3df1c501d0a19e5076304179d1e43e.socket,
>> (No such file or directory)
>>
>>
>> gluster volume status is showing the tier daemon status on all the nodes
>> as 'N', but lists the pids of a nonexistent processes.
>>
>> Just wondering if I messed up removing the tier?
>>
>> -Alastair
>>
>>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] dbench

[Dmitri Chebotarov]

Can you check if you right options enabled for the volume?
Check /var/lib/gluster/groups/virt file. Ignore shard options since you
already configured it (I think recommended shard size is 512mb)...

How is the performance on mounted glusterfs-fuse on the host?

On Thu, Oct 12, 2017 at 16:45 Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> trying to check gluster performance with dbench
>
> I'm using a replica 3 with a bonded dual gigabit (balance-alb) on all
> servers and shard (64M) enabled.
>
> I'm unable to over 3MB (three) MB/s from *inside* VM, thus I think
> there isn't any small file issue, as from inside VM there isn't any
> metadata operation for gluster.
>
> Any advice ?
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] ZFS with SSD ZIL vs XFS

[Dmitri Chebotarov]

I've had good results with using SSD as LVM cache for gluster bricks (
http://man7.org/linux/man-pages/man7/lvmcache.7.html). I still use XFS on
bricks.



On Tue, Oct 10, 2017 at 12:27 PM, Jeff Darcy  wrote:

> On Tue, Oct 10, 2017, at 11:19 AM, Gandalf Corvotempesta wrote:
> > Anyone made some performance comparison between XFS and ZFS with ZIL
> > on SSD, in gluster environment ?
> >
> > I've tried to compare both on another SDS (LizardFS) and I haven't
> > seen any tangible performance improvement.
> >
> > Is gluster different ?
>
> Probably not.  If there is, it would probably favor XFS.  The developers
> at Red Hat use XFS almost exclusively.  We at Facebook have a mix, but
> XFS is (I think) the most common.  Whatever the developers use tends to
> become "the way local filesystems work" and code is written based on
> that profile, so even without intention that tends to get a bit of a
> boost.  To the extent that ZFS makes different tradeoffs - e.g. using
> lots more memory, very different disk access patterns - it's probably
> going to have a bit more of an "impedance mismatch" with the choices
> Gluster itself has made.
>
> If you're interested in ways to benefit from a disk+SSD combo under XFS,
> it is possible to configure XFS with a separate journal device but I
> believe there were some bugs encountered when doing that.  Richard
> Wareing's upcoming Dev Summit talk on Hybrid XFS might cover those, in
> addition to his own work on using an SSD in even more interesting ways.
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] sparse files on EC volume

[Dmitri Chebotarov]

Hi Ben

Thank you.
I just ran some tests with the same data on EC and R3 volumes (same
hardware).
R3 is a lot faster

EC

untar 48.879s
find 2.993s
rm 11.244s

R3

untar 10.938s
find 0.722s
rm 4.144s



On Wed, Sep 27, 2017 at 3:12 AM, Ben Turner  wrote:

> Have you done any testing with replica 2/3?  IIRC my replica 2/3 tests out
> performed EC on smallfile workloads, it may be worth looking into if you
> can't get EC up to where you need it to be.
>
> -b
>
> - Original Message -
> > From: "Dmitri Chebotarov" <4dim...@gmail.com>
> > Cc: "gluster-users" 
> > Sent: Tuesday, September 26, 2017 9:57:55 AM
> > Subject: Re: [Gluster-users] sparse files on EC volume
> >
> > Hi Xavi
> >
> > At this time I'm using 'plain' bricks with XFS. I'll be moving to LVM
> cached
> > bricks.
> > There is no RAID for data bricks, but I'll be using hardware RAID10 for
> SSD
> > cache disks (I can use 'writeback' cache in this case).
> >
> > 'small file performance' is the main reason I'm looking at different
> options,
> > i.e. using formated sparse files.
> > I spent considerable amount of time tuning 10GB/kernel/gluster to reduce
> > latency - the small file performance improved ~50% but it's still no good
> > enough, especially when I need to use Gluster for /home folders.
> >
> > I understand limitations and single point of failure in case with sparse
> > files. I'm considering different options to provide HA
> (pacemaker/corosync,
> > keepalived or using VMs - RHEV - to deliver storage).
> >
> > Thank you for your reply.
> >
> >
> > On Tue, Sep 26, 2017 at 3:55 AM, Xavi Hernandez < jaher...@redhat.com >
> > wrote:
> >
> >
> > Hi Dmitri,
> >
> > On 22/09/17 17:07, Dmitri Chebotarov wrote:
> >
> >
> >
> > Hello
> >
> > I'm running some tests to compare performance between Gluster FUSE mount
> and
> > formated sparse files (located on the same Gluster FUSE mount).
> >
> > The Gluster volume is EC (same for both tests).
> >
> > I'm seeing HUGE difference and trying to figure out why.
> >
> > Could you explain what hardware configuration are you using ?
> >
> > Do you have a plain disk for each brick formatted in XFS, or do you have
> some
> > RAID configuration ?
> >
> >
> >
> >
> > Here is an example:
> >
> > GlusterFUSE mount:
> >
> > # cd /mnt/glusterfs
> > # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
> > 1+0 records in
> > 1+0 records out
> > 1073741824 bytes (1.1 GB) copied, 9.74757 s, *110 MB/s*
> >
> > Sparse file (located on GlusterFUSE mount):
> >
> > # truncate -l 100GB /mnt/glusterfs/xfs-100G.img
> > # mkfs.xfs /mnt/glusterfs/xfs-100G.img
> > # mount -o loop /mnt/glusterfs/xfs-100G.img /mnt/xfs-100G
> > # cd /mnt/xfs-100G
> > # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
> > 1+0 records in
> > 1+0 records out
> > 1073741824 bytes (1.1 GB) copied, 1.20576 s, *891 MB/s*
> >
> > The same goes for working with small files (i.e. code file, make, etc)
> with
> > the same data located on FUSE mount vs formated sparse file on the same
> FUSE
> > mount.
> >
> > What would explain such difference?
> >
> > First of all, doing tests with relatively small files tends to be
> misleading
> > because of caching capacity of the operating system (to minimize that,
> you
> > can add 'conv=fsync' option to dd). You should do tests with file sizes
> > bigger than the amount of physical memory on servers. This way you
> minimize
> > cache effects and see the real sustained performance.
> >
> > A second important point to note is that gluster is a distributed file
> system
> > that can be accessed simultaneously by more than one client. This means
> that
> > consistency must be assured in all cases, which makes things go to bricks
> > sooner than local filesystems normally do.
> >
> > In your case, all data saved to the fuse volume will most probably be
> present
> > on bricks once the dd command completes. On the other side, the test
> through
> > the formatted sparse file, most probably, is keeping most of the data in
> the
> > cache of the client machine.
> >
> > Note that using the formatted sparse file makes it possible a better use
> of
> > local cache, improving (relatively) small file access, but on the other
> > side

Re: [Gluster-users] sparse files on EC volume

[Dmitri Chebotarov]

Hi Xavi

At this time I'm using 'plain' bricks with XFS. I'll be moving to LVM
cached bricks.
There is no RAID for data bricks, but I'll be using hardware RAID10 for SSD
cache disks (I can use 'writeback' cache in this case).

'small file performance' is the main reason I'm looking at different
options, i.e. using formated sparse files.
I spent considerable amount of time tuning 10GB/kernel/gluster to reduce
latency - the small file performance improved ~50% but it's still no good
enough, especially when I need to use Gluster for /home folders.

I understand limitations and single point of failure in case with sparse
files. I'm considering different options to provide HA (pacemaker/corosync,
keepalived or using VMs - RHEV - to deliver storage).

Thank you for your reply.


On Tue, Sep 26, 2017 at 3:55 AM, Xavi Hernandez  wrote:

> Hi Dmitri,
>
> On 22/09/17 17:07, Dmitri Chebotarov wrote:
>
>>
>> Hello
>>
>> I'm running some tests to compare performance between Gluster FUSE mount
>> and formated sparse files (located on the same Gluster FUSE mount).
>>
>> The Gluster volume is EC (same for both tests).
>>
>> I'm seeing HUGE difference and trying to figure out why.
>>
>
> Could you explain what hardware configuration are you using ?
>
> Do you have a plain disk for each brick formatted in XFS, or do you have
> some RAID configuration ?
>
>
>> Here is an example:
>>
>> GlusterFUSE mount:
>>
>> # cd /mnt/glusterfs
>> # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
>> 1+0 records in
>> 1+0 records out
>> 1073741824 bytes (1.1 GB) copied, 9.74757 s, *110 MB/s*
>>
>> Sparse file (located on GlusterFUSE mount):
>>
>> # truncate -l 100GB /mnt/glusterfs/xfs-100G.img
>> # mkfs.xfs /mnt/glusterfs/xfs-100G.img
>> # mount -o loop /mnt/glusterfs/xfs-100G.img /mnt/xfs-100G
>> # cd /mnt/xfs-100G
>> # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
>> 1+0 records in
>> 1+0 records out
>> 1073741824 bytes (1.1 GB) copied, 1.20576 s, *891 MB/s*
>>
>> The same goes for working with small files (i.e. code file, make, etc)
>> with the same data located on FUSE mount vs formated sparse file on the
>> same FUSE mount.
>>
>> What would explain such difference?
>>
>
> First of all, doing tests with relatively small files tends to be
> misleading because of caching capacity of the operating system (to minimize
> that, you can add 'conv=fsync' option to dd). You should do tests with file
> sizes bigger than the amount of physical memory on servers. This way you
> minimize cache effects and see the real sustained performance.
>
> A second important point to note is that gluster is a distributed file
> system that can be accessed simultaneously by more than one client. This
> means that consistency must be assured in all cases, which makes things go
> to bricks sooner than local filesystems normally do.
>
> In your case, all data saved to the fuse volume will most probably be
> present on bricks once the dd command completes. On the other side, the
> test through the formatted sparse file, most probably, is keeping most of
> the data in the cache of the client machine.
>
> Note that using the formatted sparse file makes it possible a better use
> of local cache, improving (relatively) small file access, but on the other
> side, this filesystem can only be used from a single client (single mount).
> If this client fails for some reason, you will loose access to your data.
>
>
>> How does Gluster work with sparse files in general? I may move some of
>> the data on gluster volumes to formated sparse files..
>>
>
> Gluster works fine with sparse files. However you should consider the
> previous points before choosing the formatted sparse files option. I guess
> that the sustained throughput will be very similar for bigger files.
>
> Regards,
>
> Xavi
>
>
>> Thank you.
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] EC 1+2

[Dmitri Chebotarov]

Hi

Take a look at this link (under “Optimal volumes”), for Erasure Coded
volume optimal configuration

http://docs.gluster.org/Administrator%20Guide/Setting%20Up%20Volumes/

On Sat, Sep 23, 2017 at 10:01 Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:

> Is possible to create a dispersed volume 1+2 ? (Almost the same as replica
> 3, the same as RAID-6)
>
> If yes, how many server I have to add in the future to expand the storage?
> 1 or 3?
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] sparse files on EC volume

[Dmitri Chebotarov]

Hello

I'm running some tests to compare performance between Gluster FUSE mount
and formated sparse files (located on the same Gluster FUSE mount).

The Gluster volume is EC (same for both tests).

I'm seeing HUGE difference and trying to figure out why.

Here is an example:

GlusterFUSE mount:

# cd /mnt/glusterfs
# rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 9.74757 s, *110 MB/s*

Sparse file (located on GlusterFUSE mount):

# truncate -l 100GB /mnt/glusterfs/xfs-100G.img
# mkfs.xfs /mnt/glusterfs/xfs-100G.img
# mount -o loop /mnt/glusterfs/xfs-100G.img /mnt/xfs-100G
# cd /mnt/xfs-100G
# rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 1.20576 s, *891 MB/s*

The same goes for working with small files (i.e. code file, make, etc) with
the same data located on FUSE mount vs formated sparse file on the same
FUSE mount.

What would explain such difference?

How does Gluster work with sparse files in general? I may move some of the
data on gluster volumes to formated sparse files..

Thank you.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow performance of gluster volume

[Dmitri Chebotarov]

Hi Abi

Can you please share your current transfer speeds after you made the change?

Thank you.

On Mon, Sep 11, 2017 at 9:55 AM, Ben Turner  wrote:

> - Original Message -
> > From: "Abi Askushi" 
> > To: "Ben Turner" 
> > Cc: "Krutika Dhananjay" , "gluster-user" <
> gluster-users@gluster.org>
> > Sent: Monday, September 11, 2017 1:40:42 AM
> > Subject: Re: [Gluster-users] Slow performance of gluster volume
> >
> > Did not upgrade yet gluster. I am still  using 3.8.12. Only the mentioned
> > changes did provide the performance boost.
> >
> > From which version to which version did you see such performance boost? I
> > will try to upgrade and check difference also.
>
> Unfortunately I didn't record the package versions, I also may have done
> the same thing as you :)
>
> -b
>
> >
> > On Sep 11, 2017 2:45 AM, "Ben Turner"  wrote:
> >
> > Great to hear!
> >
> > - Original Message -
> > > From: "Abi Askushi" 
> > > To: "Krutika Dhananjay" 
> > > Cc: "gluster-user" 
> > > Sent: Friday, September 8, 2017 7:01:00 PM
> > > Subject: Re: [Gluster-users] Slow performance of gluster volume
> > >
> > > Following changes resolved the perf issue:
> > >
> > > Added the option
> > > /etc/glusterfs/glusterd.vol :
> > > option rpc-auth-allow-insecure on
> >
> > Was it this setting or was it the gluster upgrade, do you know for sure?
> > It may be helpful to others to know for sure(Im interested too:).
> >
> > -b
> >
> > >
> > > restarted glusterd
> > >
> > > Then set the volume option:
> > > gluster volume set vms server.allow-insecure on
> > >
> > > I am reaching now the max network bandwidth and performance of VMs is
> > quite
> > > good.
> > >
> > > Did not upgrade the glusterd.
> > >
> > > As a next try I am thinking to upgrade gluster to 3.12 + test libgfapi
> > > integration of qemu by upgrading to ovirt 4.1.5 and check vm perf.
> > >
> > >
> > > On Sep 6, 2017 1:20 PM, "Abi Askushi" < rightkickt...@gmail.com >
> wrote:
> > >
> > >
> > >
> > > I tried to follow step from
> > > https://wiki.centos.org/SpecialInterestGroup/Storage to install latest
> > > gluster on the first node.
> > > It installed 3.10 and not 3.11. I am not sure how to install 3.11
> without
> > > compiling it.
> > > Then when tried to start the gluster on the node the bricks were
> reported
> > > down (the other 2 nodes have still 3.8). No sure why. The logs were
> > showing
> > > the below (even after rebooting the server):
> > >
> > > [2017-09-06 10:56:09.023777] E [rpcsvc.c:557:rpcsvc_check_
> and_reply_error]
> > > 0-rpcsvc: rpc actor failed to complete successfully
> > > [2017-09-06 10:56:09.024122] E [server-helpers.c:395:server_
> alloc_frame]
> > > (-->/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x325)
> [0x7f2d0ec20905]
> > > -->/usr/lib64/glusterfs/3.10.5/xlator/protocol/server.so(+0x3006b)
> > > [0x7f2cfa4bf06b]
> > > -->/usr/lib64/glusterfs/3.10.5/xlator/protocol/server.so(+0xdb34)
> > > [0x7f2cfa49cb34] ) 0-server: invalid argument: client [Invalid
> argument]
> > >
> > > Do I need to upgrade all nodes before I attempt to start the gluster
> > > services?
> > > I reverted the first node back to 3.8 at the moment and all restored.
> > > Also tests with eager lock disabled did not make any difference.
> > >
> > >
> > >
> > >
> > > On Wed, Sep 6, 2017 at 11:15 AM, Krutika Dhananjay <
> kdhan...@redhat.com >
> > > wrote:
> > >
> > >
> > >
> > > Do you see any improvement with 3.11.1 as that has a patch that
> improves
> > perf
> > > for this kind of a workload
> > >
> > > Also, could you disable eager-lock and check if that helps? I see that
> max
> > > time is being spent in acquiring locks.
> > >
> > > -Krutika
> > >
> > > On Wed, Sep 6, 2017 at 1:38 PM, Abi Askushi < rightkickt...@gmail.com
> >
> > > wrote:
> > >
> > >
> > >
> > > Hi Krutika,
> > >
> > > Is it anything in the profile indicating what is causing this
> bottleneck?
> > In
> > > case i can collect any other info let me know.
> > >
> > > Thanx
> > >
> > > On Sep 5, 2017 13:27, "Abi Askushi" < rightkickt...@gmail.com > wrote:
> > >
> > >
> > >
> > > Hi Krutika,
> > >
> > > Attached the profile stats. I enabled profiling then ran some dd tests.
> > Also
> > > 3 Windows VMs are running on top this volume but did not do any stress
> > > testing on the VMs. I have left the profiling enabled in case more
> time is
> > > needed for useful stats.
> > >
> > > Thanx
> > >
> > > On Tue, Sep 5, 2017 at 12:48 PM, Krutika Dhananjay <
> kdhan...@redhat.com >
> > > wrote:
> > >
> > >
> > >
> > > OK my understanding is that with preallocated disks the performance
> with
> > and
> > > without shard will be the same.
> > >
> > > In any case, please attach the volume profile[1], so we can see what
> else
> > is
> > > slowing things down.
> > >
> > > -Krutika
> > >
> > > [1] -
> > > https://gluster.readthedocs.io/en/latest/Administrator%
> > 20Guide/Monitoring%20Workload/#running-glusterfs-volume-profile-command
> > >
> > > On Tue, Sep 5, 2017 at 2:32 PM, Abi Askushi < rightkickt

Re: [Gluster-users] Hot Tier

[Dmitri Chebotarov]

Hello

I reattached hot tier to a new empty EC volume and started to copy data to
the volume.
Good news is I can see files now on SSD bricks (hot tier) - 'find
/path/to/brick -type f' shows files, before 'find' would only show dirs.

But I've got a 'rebalance' error in glusterd.log file after I attached hot
tier.

[2017-08-02 14:09:01.489891] E [MSGID: 106062]
[glusterd-utils.c:9182:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
failed to get index
The message "E [MSGID: 106062]
[glusterd-utils.c:9182:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd:
failed to get index" repeated 10 times between [2017-08-02 14:09:01.489891]
and [2017-08-02 14:09:01.545027]

This is output from 'rebalance status' command:

# gluster volume rebalance voldata3 status
Node Rebalanced-files  size
  scanned  failures   skipped   status  run time in
h:m:s
   -  ---   ---
---   ---   --- 
--
   localhost00Bytes
0 0 0  in progress0:0:0
   GFSRV1800Bytes
  0 0 0  in progress0:0:0
   GFSRV2000Bytes
  0 0 0  in progress0:0:0
   GFSRV2100Bytes
  0 0 0  in progress0:0:0
   GFSRV2300Bytes
  0 0 0  in progress0:0:0
   GFSRV1700Bytes
  0 0 0  in progress0:0:0
   GFSRV2400Bytes
  0 0 0  in progress0:0:0
   GFSRV1600Bytes
  0 0 0  in progress0:0:0
   GFSRV1500Bytes
  0 0 0  in progress0:0:0
   GFSRV1400Bytes
  0 0 0  in progress0:0:0
   GFSRV2200Bytes
  0 0 0  in progress0:0:0
   GFSRV1900Bytes
  0 0 0  in progress0:0:0
volume rebalance: voldata3: success

and 'tier status' output:

# gluster volume tier voldata3 status
Node Promoted files   Demoted filesStatus
----
localhost00in progress
GFSRV1800in progress
GFSRV2000in progress
GFSRV2100in progress
GFSRV2300in progress
GFSRV1700in progress
GFSRV2400in progress
GFSRV1600in progress
GFSRV1500in progress
GFSRV1400in progress
GFSRV2200in progress
GFSRV1900in progress
Tiering Migration Functionality: voldata3: success

'vol status' shows one active task:

Task Status of Volume voldata3
--
Task : Tier migration
ID   : c4c33b04-2a1e-4e53-b1f5-a96ec6d9d851
Status   : in progress


No errors reported in 'voldata3-tier-.log' file.

I'll keep monitoring it for few day. I expect to see some 'cooled' data
moving to 'cold tier'.

Thank you.

On Tue, Aug 1, 2017 at 1:32 AM, Hari Gowtham  wrote:

> Hi,
>
> You have missed the log files.
>
> Can you attach them?
>
>
> On Mon, Jul 31, 2017 at 7:22 PM, Dmitri Chebotarov <4dim...@gmail.com>
> wrote:
> > Hi
> >
> > At this point I already detached Hot Tier volume to run rebalance. Many
> > volume settings only take effect for 

[Gluster-users] RECOMMENDED CONFIGURATIONS - DISPERSED VOLUME

[Dmitri Chebotarov]

Hi

I'm looking for an advise to configure a dispersed volume.
I have 12 servers and would like to use 10:2 ratio.

Yet RH recommends 8:3 or 8:4 in this case:

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/chap-Recommended-Configuration_Dispersed.html

My goal is to create 2PT volume, and going with 10:2 vs 8:3/4 saves a few
bricks. With 10:2 I'll use 312 8TB bricks and with 8:3 it's 396 8TB bricks
(36 8:3 slices to evenly distribute between all servers/bricks)

As I see it 8:3/4 vs 10:2 gives more data redundancy (3 servers vs 2
servers can be offline), but is critical with 12 nodes? Nodes are new and
under warranty, it's unlikely I will lose 3 servers  at the same time (10:2
goes offline). Or should I follow RH recommended configuration and use
8:3/4?

Thank you.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gluster volume 3.10.4 hangs

[Dmitri Chebotarov]

Hi

With only two nodes it's recommended to set
cluster.server-quorum-type=server and cluster.server-quorum-ratio=51% (i.e.
more than 50%).

On Mon, Jul 31, 2017 at 4:12 AM, Seva Gluschenko  wrote:

> Hi folks,
>
>
> I'm running a simple gluster setup with a single volume replicated at two
> servers, as follows:
>
> Volume Name: gv0
> Type: Replicate
> Volume ID: dd4996c0-04e6-4f9b-a04e-73279c4f112b
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: sst0:/var/glusterfs
> Brick2: sst2:/var/glusterfs
> Options Reconfigured:
> cluster.self-heal-daemon: enable
> performance.readdir-ahead: on
> nfs.disable: on
> transport.address-family: inet
>
> This volume is used to store data in highload production, and recently I
> faced two major problems that made the whole idea of using gluster quite
> questionnable, so I would like to ask gluster developers and/or call for
> community wisdom in hope that I might be missing something. The problem is,
> when it happened that one of replica servers hung, it caused the whole
> glusterfs to hang. Could you please drop me a hint, is it expected
> behaviour, or are there any tweaks and server or volume settings that might
> be altered to change this? Any help would be appreciated much.
>
>
> --
> Best Regards,
>
> Seva Gluschenko
> CTO @ http://webkontrol.ru
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Hot Tier

[Dmitri Chebotarov]

Hi

At this point I already detached Hot Tier volume to run rebalance. Many
volume settings only take effect for the new data (or rebalance), so I
thought may this was the case with Hot Tier as well. Once rebalance
finishes, I'll re-attache hot tier.

cluster.write-freq-threshold and cluster.read-freq-threshold control number
of times data is read/write before it moved to hot tier. In my case both
are set to '2', I didn't think I needed to disable
performance.io-cache/quick-read as well. Will give it a try.

Here is the volume info (no hot tier at this time)

~]# gluster v info home

Volume Name: home
Type: Disperse
Volume ID: 4583a3cf-4deb-4707-bd0d-e7defcb1c39b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (8 + 4) = 12
Transport-type: tcp
Bricks:
Brick1: MMR01:/rhgs/b0/data
Brick2: MMR02:/rhgs/b0/data
Brick3: MMR03:/rhgs/b0/data
Brick4: MMR04:/rhgs/b0/data
Brick5: MMR05:/rhgs/b0/data
Brick6: MMR06:/rhgs/b0/data
Brick7: MMR07:/rhgs/b0/data
Brick8: MMR08:/rhgs/b0/data
Brick9: MMR09:/rhgs/b0/data
Brick10: MMR10:/rhgs/b0/data
Brick11: MMR11:/rhgs/b0/data
Brick12: MMR12:/rhgs/b0/data
Options Reconfigured:
diagnostics.client-log-level: CRITICAL
cluster.write-freq-threshold: 2
cluster.read-freq-threshold: 2
features.record-counters: on
nfs.disable: on
performance.readdir-ahead: enable
transport.address-family: inet
client.event-threads: 4
server.event-threads: 4
cluster.lookup-optimize: on
cluster.readdir-optimize: on
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 1
cluster.data-self-heal-algorithm: full
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 5
performance.write-behind-window-size: 1MB
performance.client-io-threads: on
performance.read-ahead: disable
performance.cache-size: 256MB
performance.io-thread-count: 16
performance.strict-o-direct: on
network.ping-timeout: 30
network.remote-dio: disable
user.cifs: off
features.quota: on
features.inode-quota: on
features.quota-deem-statfs: on

~]# gluster v get home  performance.io-cache
performance.io-cacheon

~]# gluster v get home  performance.quick-read
performance.quick-read  on

Thank you.

On Mon, Jul 31, 2017 at 5:16 AM, Hari Gowtham  wrote:

> Hi,
>
> Before you try turning off the perf translators can you send us the
> following,
> So we will make sure that the other things haven't gone wrong.
>
> can you send us the log files for tier (would be better if you attach
> other logs too),
> the version of gluster you are using, the client, and the output for:
> gluster v info
> gluster v get v1 performance.io-cache
> gluster v get v1 performance.quick-read
>
> Do send us this and then we will let you know what should be done,
> as reads should also cause promotion
>
>
> On Mon, Jul 31, 2017 at 2:21 PM, Hari Gowtham  wrote:
> > For the tier daemon to migrate the files for read, few performance
> > translators have to be turned off.
> > By default the performance quick-read and io-cache are turned on. You
> > can turn them off so that
> > the files will be migrated for read.
> >
> > On Mon, Jul 31, 2017 at 11:34 AM, Hari Gowtham 
> wrote:
> >> Hi,
> >>
> >> If it was just reads then the tier daemon won't migrate the files to
> hot tier.
> >> If you create a file or write to a file that file will be made
> >> available on the hot tier.
> >>
> >>
> >> On Mon, Jul 31, 2017 at 11:06 AM, Nithya Balachandran
> >>  wrote:
> >>> Milind and Hari,
> >>>
> >>> Can you please take a look at this?
> >>>
> >>> Thanks,
> >>> Nithya
> >>>
> >>> On 31 July 2017 at 05:12, Dmitri Chebotarov <4dim...@gmail.com> wrote:
> >>>>
> >>>> Hi
> >>>>
> >>>> I'm looking for an advise on hot tier feature - how can I tell if the
> hot
> >>>> tier is working?
> >>>>
> >>>> I've attached replicated-distributed hot tier to an EC volume.
> >>>> Yet, I don't think it's working, at least I don't see any files
> directly
> >>>> on the bricks (only folder structure). 'Status' command has all 0s
> and 'In
> >>>> progress' for all servers.
> >>>>
> >>>> ~]# gluster volume tier home status
> >>>> Node Promoted files   Demoted filesStatus
> >>>> ---
> -
> >>>> localhost0

[Gluster-users] Hot Tier

[Dmitri Chebotarov]

Hi

I'm looking for an advise on hot tier feature - how can I tell if the hot
tier is working?

I've attached replicated-distributed hot tier to an EC volume.
Yet, I don't think it's working, at least I don't see any files directly on
the bricks (only folder structure). 'Status' command has all 0s and 'In
progress' for all servers.

~]# gluster volume tier home status
Node Promoted files   Demoted filesStatus
----
localhost00in progress
MMR1100in progress
MMR0800in progress
MMR0300in progress
MMR0200in progress
MMR0700in progress
MMR0600in progress
MMR0900in progress
MMR1200in progress
MMR1000in progress
MMR0500in progress
MMR0400in progress
Tiering Migration Functionality: home: success


I have a folder with .yml files (Ansible) on the gluster volume, which as I
understand is 'cache friendly'.
No matter how many times I read files, nothing is moved to the hot tier
bricks.

Thank you.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users