Re: [Gluster-users] VM going down

[Lindsay Mathieson]

 On 11/05/2017 9:51 AM, Alessandro Briosi wrote:

On one it reports about Leaked clusters but I don't think this might cause
the problem (or not?)


Should be fine

-- 
Lindsay Mathieson
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] VM going down

[Alessandro Briosi]

Il 09/05/2017 23:41, Lindsay Mathieson ha scritto:

On 10/05/2017 12:59 AM, Alessandro Briosi wrote:

Also the seek errors where there before when there was no arbiter
(only 2 replica).
And finally seek error is triggered when the VM is started (at least
the one in the logs).


Could there be a corruption problem in the qcow2 image? have you run
"qemu-img check" against it?



Not sure about this. But does not seems so.
The qemu-img check reports everything is ok with disks.

root@srvpve1:/mnt/pve/datastore2/images/201# qemu-img check 
vm-201-disk-1.qcow2

No errors were found on the image.
655360/655360 = 100.00% allocated, 1.11% fragmented, 0.00% compressed 
clusters

Image end offset: 42969661440


On one it reports about Leaked clusters but I don't think this might 
cause the problem (or not?)


root@srvpve1:/mnt/pve/datastore1/images/101# qemu-img check 
vm-101-disk-1.qcow2

Leaked cluster 409006 refcount=1 reference=0
Leaked cluster 624338 refcount=1 reference=0
Leaked cluster 791103 refcount=1 reference=0

3 leaked clusters were found on the image.
This means waste of disk space, but no harm to data.
8192000/8192000 = 100.00% allocated, 1.24% fragmented, 0.00% compressed 
clusters

Image end offset: 539424129024

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow write times to gluster disk

[Pat Haley]

Hi Pranith,

Since we are mounting the partitions as the bricks, I tried the dd test 
writing to /.glusterfs/. The 
results without oflag=sync were 1.6 Gb/s (faster than gluster but not as 
fast as I was expecting given the 1.2 Gb/s to the no-gluster area w/ 
fewer disks).


Pat


On 05/10/2017 01:27 PM, Pranith Kumar Karampuri wrote:



On Wed, May 10, 2017 at 10:15 PM, Pat Haley > wrote:



Hi Pranith,

Not entirely sure (this isn't my area of expertise). I'll run your
answer by some other people who are more familiar with this.

I am also uncertain about how to interpret the results when we
also add the dd tests writing to the /home area (no gluster, still
on the same machine)

  * dd test without oflag=sync (rough average of multiple tests)
  o gluster w/ fuse mount : 570 Mb/s
  o gluster w/ nfs mount:  390 Mb/s
  o nfs (no gluster):  1.2 Gb/s
  * dd test with oflag=sync (rough average of multiple tests)
  o gluster w/ fuse mount:  5 Mb/s
  o gluster w/ nfs mount:  200 Mb/s
  o nfs (no gluster): 20 Mb/s

Given that the non-gluster area is a RAID-6 of 4 disks while each
brick of the gluster area is a RAID-6 of 32 disks, I would naively
expect the writes to the gluster area to be roughly 8x faster than
to the non-gluster.


I think a better test is to try and write to a file using nfs without 
any gluster to a location that is not inside the brick but someother 
location that is on same disk(s). If you are mounting the partition as 
the brick, then we can write to a file inside .glusterfs directory, 
something like /.glusterfs/.



I still think we have a speed issue, I can't tell if fuse vs nfs
is part of the problem.


I got interested in the post because I read that fuse speed is lesser 
than nfs speed which is counter-intuitive to my understanding. So 
wanted clarifications. Now that I got my clarifications where fuse 
outperformed nfs without sync, we can resume testing as described 
above and try to find what it is. Based on your email-id I am guessing 
you are from Boston and I am from Bangalore so if you are okay with 
doing this debugging for multiple days because of timezones, I will be 
happy to help. Please be a bit patient with me, I am under a release 
crunch but I am very curious with the problem you posted.


  Was there anything useful in the profiles?


Unfortunately profiles didn't help me much, I think we are collecting 
the profiles from an active volume, so it has a lot of information 
that is not pertaining to dd so it is difficult to find the 
contributions of dd. So I went through your post again and found 
something I didn't pay much attention to earlier i.e. oflag=sync, so 
did my own tests on my setup with FUSE so sent that reply.



Pat



On 05/10/2017 12:15 PM, Pranith Kumar Karampuri wrote:

Okay good. At least this validates my doubts. Handling O_SYNC in
gluster NFS and fuse is a bit different.
When application opens a file with O_SYNC on fuse mount then each
write syscall has to be written to disk as part of the syscall
where as in case of NFS, there is no concept of open. NFS
performs write though a handle saying it needs to be a
synchronous write, so write() syscall is performed first then it
performs fsync(). so an write on an fd with O_SYNC becomes
write+fsync. I am suspecting that when multiple threads do this
write+fsync() operation on the same file, multiple writes are
batched together to be written do disk so the throughput on the
disk is increasing is my guess.

Does it answer your doubts?

On Wed, May 10, 2017 at 9:35 PM, Pat Haley > wrote:


Without the oflag=sync and only a single test of each, the
FUSE is going faster than NFS:

FUSE:
mseas-data2(dri_nascar)% dd if=/dev/zero count=4096
bs=1048576 of=zeros.txt conv=sync
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 7.46961 s, 575 MB/s


NFS
mseas-data2(HYCOM)% dd if=/dev/zero count=4096 bs=1048576
of=zeros.txt conv=sync
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 11.4264 s, 376 MB/s



On 05/10/2017 11:53 AM, Pranith Kumar Karampuri wrote:

Could you let me know the speed without oflag=sync on both
the mounts? No need to collect profiles.

On Wed, May 10, 2017 at 9:17 PM, Pat Haley > wrote:


Here is what I see now:

[root@mseas-data2 ~]# gluster volume info

Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
  

Re: [Gluster-users] Quota limits gone after upgrading to 3.8

[mabi]

Hi Sanoj,

I do understand that my quotas are still working on my GluserFS volume but not 
displayed in the output of the volume quota list command.

I now did the test of re-adding the quota by running for example:

gluster volume quota myvolume limit-usage /directoryX 50GB

After that I ran the volume quota list command and luckily enough my quota is 
available display again in the list. So I guess I will re-add the quotas so 
that the are displayed again in the list. That's the easiest way for me but I 
do hope the quotas stay next time I upgrade...

Regards,
M.

 Original Message 
Subject: Re: [Gluster-users] Quota limits gone after upgrading to 3.8
Local Time: May 10, 2017 8:48 AM
UTC Time: May 10, 2017 6:48 AM
From: sunni...@redhat.com
To: mabi 
Gluster Users 

Hi Mabi,

Note that limits are still configured and working.
re-adding the limits will not help here (unless you are willing to disable and 
re-enable quota first).
The reason is if a gfid exists in quota.conf (because a limit was earlier set 
on it), it does not need change when limit changes.

The quota.conf file only keep track of which gfid have limit set. The original 
value of the limits are set in xattr on filesystem
Another work around without mauallly touching quota.conf is,
> Create a new dummy directory anywhere in the FS. add a limit in this 
> directory.

After this you should be able to see the listing.
If you remove this dummy directory or limit on it, you will once again be 
exposed to same issue.

Regards,
Sanoj

On Tue, May 9, 2017 at 10:59 PM, mabi  wrote:
Hi Sanoj,

Thanks for pointing me at this bug, I was not aware about it.

As this is a production GlusterFS cluster I would rather not mess with the 
quota.conf file as you suggested. Instead I will simply re-add all my quotas by 
running the following command again:

gluster volume quota myvolume limit-usage /directory1 100GB

Can you confirm me that this is safe to run again?

As soon as I have a minute I will complete your survey about quotas.

Best,
M.

 Original Message 
Subject: Re: [Gluster-users] Quota limits gone after upgrading to 3.8
Local Time: May 9, 2017 6:50 AM
UTC Time: May 9, 2017 4:50 AM
From: sunni...@redhat.com
To: mabi 
Gluster Users 

Hi mabi,

This bug was fixed recently, 
https://bugzilla.redhat.com/show_bug.cgi?id=1414346. It would be available in 
3.11 release. I will plan to back port same to earlier releases.

Your quota limits are still set and honored, It is only the listing that has 
gone wrong. Using list with command with single path should display the limit 
on that path. The printing of list gets messed up when the last gfid in the 
quota.conf file is not present in the FS (due to an rmdir without a remove 
limit)

You could use the following workaround to get rid of the issue.
=> Remove exactly the last 17 bytes of " 
/var/lib/glusterd/vols//quota.conf"

Note: keep a backup of quota.conf for safety
If this does not solve the issue, please revert back with
1) quota.conf file
2) output of list command (when executed along with path)

3) getfattr -d -m . -e hex  | grep limit
It would be great to have your feedback for quota on this thread 
(http://lists.gluster.org/pipermail/gluster-users/2017-April/030676.html)

Thanks & Regards,
Sanoj

On Mon, May 8, 2017 at 7:58 PM, mabi  wrote:
Hello,

I upgraded last week my 2 nodes replica GlusterFS cluster from 3.7.20 to 3.8.11 
and on one of the volumes I use the quota feature of GlusterFS. Unfortunately, 
I just noticed by using the usual command "gluster volume quota myvolume list" 
that all my quotas on that volume are gone. I had around 10 different quotas 
set on different directories.

Does anyone have an idea where the quotas have vanished? are they gone for 
always and do I need to re-set them all?

Regards,
M.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Is there difference when Nfs-Ganesha is unavailable

[ML Wong]

Soumya,
I should have mentioned in my first email. The VIP was always able to
failover to the remaining nodes.  But in many of my testings, the failover
IP just did not carry over the states for the NFS client. So, it always
look like  the NFS server is unavailable.

Thanks for your response.  Any pointers on where to look will be great.
Lately, I also found out different NFS client played a significant role in
my testings also, unfortunately...


On Tue, May 9, 2017 at 11:21 PM Soumya Koduri  wrote:

>
>
> On 05/10/2017 04:18 AM, ML Wong wrote:
> > While I m troubleshooting the failover of Nfs-Ganesha, the failover is
> > always successful when I shutdown Nfs-Ganesha service online while the
> > OS is running. However, it always failed when I did a either shutdown -r
> > or power-reset.
> >
> > During the failure, the Nfs client was just hung. Like you could not do
> > a "df" or "ls" of the mount point. The share will eventually failover to
> > the remaining expected node usually after 15 - 20 minutes.
>
> The time taken by pacemaker/corosync services to determine if a node is
> down is usually longer compared to the service down case. But yes it
> should n't take more than couple of minutes.
>
> Could you please check (may be by constantly querying) on how long it
> takes for the virtual-IP to failover by using either 'pcs status' or 'ip
> a' commands. If the IP failover happens quickly but if its just the NFS
> clients taking time to respond, then we have added usage of portblock
> feature to speed up client re-connects post failover. The fixes are
> available (from release-3.9). But before upgrading I suggest to check if
> the delay is with IP failover or client reconnects post failover.
>
> Thanks,
> Soumya
>
> >
> > Running on Centos7, gluster 3.7.1x, Nfs-Ganesha 2.3.0.x. I currently
> > don't have the resources to upgrade, but if all of experts here think
> > that's the only route. I guess I will have to make a case ...
> >
> > Thanks in advance!
> >
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> >
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow write times to gluster disk

[Pranith Kumar Karampuri]

On Wed, May 10, 2017 at 10:15 PM, Pat Haley  wrote:

>
> Hi Pranith,
>
> Not entirely sure (this isn't my area of expertise).  I'll run your answer
> by some other people who are more familiar with this.
>
> I am also uncertain about how to interpret the results when we also add
> the dd tests writing to the /home area (no gluster, still on the same
> machine)
>
>- dd test without oflag=sync (rough average of multiple tests)
>- gluster w/ fuse mount : 570 Mb/s
>   - gluster w/ nfs mount:  390 Mb/s
>   - nfs (no gluster):  1.2 Gb/s
>- dd test with oflag=sync (rough average of multiple tests)
>   - gluster w/ fuse mount:  5 Mb/s
>   - gluster w/ nfs mount:  200 Mb/s
>   - nfs (no gluster): 20 Mb/s
>
> Given that the non-gluster area is a RAID-6 of 4 disks while each brick of
> the gluster area is a RAID-6 of 32 disks, I would naively expect the writes
> to the gluster area to be roughly 8x faster than to the non-gluster.
>

I think a better test is to try and write to a file using nfs without any
gluster to a location that is not inside the brick but someother location
that is on same disk(s). If you are mounting the partition as the brick,
then we can write to a file inside .glusterfs directory, something like
/.glusterfs/.


> I still think we have a speed issue, I can't tell if fuse vs nfs is part
> of the problem.
>

I got interested in the post because I read that fuse speed is lesser than
nfs speed which is counter-intuitive to my understanding. So wanted
clarifications. Now that I got my clarifications where fuse outperformed
nfs without sync, we can resume testing as described above and try to find
what it is. Based on your email-id I am guessing you are from Boston and I
am from Bangalore so if you are okay with doing this debugging for multiple
days because of timezones, I will be happy to help. Please be a bit patient
with me, I am under a release crunch but I am very curious with the problem
you posted.

  Was there anything useful in the profiles?
>

Unfortunately profiles didn't help me much, I think we are collecting the
profiles from an active volume, so it has a lot of information that is not
pertaining to dd so it is difficult to find the contributions of dd. So I
went through your post again and found something I didn't pay much
attention to earlier i.e. oflag=sync, so did my own tests on my setup with
FUSE so sent that reply.


>
> Pat
>
>
>
> On 05/10/2017 12:15 PM, Pranith Kumar Karampuri wrote:
>
> Okay good. At least this validates my doubts. Handling O_SYNC in gluster
> NFS and fuse is a bit different.
> When application opens a file with O_SYNC on fuse mount then each write
> syscall has to be written to disk as part of the syscall where as in case
> of NFS, there is no concept of open. NFS performs write though a handle
> saying it needs to be a synchronous write, so write() syscall is performed
> first then it performs fsync(). so an write on an fd with O_SYNC becomes
> write+fsync. I am suspecting that when multiple threads do this
> write+fsync() operation on the same file, multiple writes are batched
> together to be written do disk so the throughput on the disk is increasing
> is my guess.
>
> Does it answer your doubts?
>
> On Wed, May 10, 2017 at 9:35 PM, Pat Haley  wrote:
>
>>
>> Without the oflag=sync and only a single test of each, the FUSE is going
>> faster than NFS:
>>
>> FUSE:
>> mseas-data2(dri_nascar)% dd if=/dev/zero count=4096 bs=1048576
>> of=zeros.txt conv=sync
>> 4096+0 records in
>> 4096+0 records out
>> 4294967296 bytes (4.3 GB) copied, 7.46961 s, 575 MB/s
>>
>>
>> NFS
>> mseas-data2(HYCOM)% dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt
>> conv=sync
>> 4096+0 records in
>> 4096+0 records out
>> 4294967296 bytes (4.3 GB) copied, 11.4264 s, 376 MB/s
>>
>>
>>
>> On 05/10/2017 11:53 AM, Pranith Kumar Karampuri wrote:
>>
>> Could you let me know the speed without oflag=sync on both the mounts? No
>> need to collect profiles.
>>
>> On Wed, May 10, 2017 at 9:17 PM, Pat Haley  wrote:
>>
>>>
>>> Here is what I see now:
>>>
>>> [root@mseas-data2 ~]# gluster volume info
>>>
>>> Volume Name: data-volume
>>> Type: Distribute
>>> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>>> Status: Started
>>> Number of Bricks: 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: mseas-data2:/mnt/brick1
>>> Brick2: mseas-data2:/mnt/brick2
>>> Options Reconfigured:
>>> diagnostics.count-fop-hits: on
>>> diagnostics.latency-measurement: on
>>> nfs.exports-auth-enable: on
>>> diagnostics.brick-sys-log-level: WARNING
>>> performance.readdir-ahead: on
>>> nfs.disable: on
>>> nfs.export-volumes: off
>>>
>>>
>>>
>>> On 05/10/2017 11:44 AM, Pranith Kumar Karampuri wrote:
>>>
>>> Is this the volume info you have?
>>>
>>> >* [root at mseas-data2 
>>> > ~]# gluster volume 
>>> >info
>>> *>>* Volume Name: data-volume
>>> *>* Type: Distribute
>>> 

Re: [Gluster-users] Slow write times to gluster disk

[Pat Haley]

Hi Pranith,

Not entirely sure (this isn't my area of expertise).  I'll run your 
answer by some other people who are more familiar with this.


I am also uncertain about how to interpret the results when we also add 
the dd tests writing to the /home area (no gluster, still on the same 
machine)


 * dd test without oflag=sync (rough average of multiple tests)
 o gluster w/ fuse mount : 570 Mb/s
 o gluster w/ nfs mount:  390 Mb/s
 o nfs (no gluster):  1.2 Gb/s
 * dd test with oflag=sync (rough average of multiple tests)
 o gluster w/ fuse mount:  5 Mb/s
 o gluster w/ nfs mount:  200 Mb/s
 o nfs (no gluster): 20 Mb/s

Given that the non-gluster area is a RAID-6 of 4 disks while each brick 
of the gluster area is a RAID-6 of 32 disks, I would naively expect the 
writes to the gluster area to be roughly 8x faster than to the non-gluster.


I still think we have a speed issue, I can't tell if fuse vs nfs is part 
of the problem.  Was there anything useful in the profiles?


Pat


On 05/10/2017 12:15 PM, Pranith Kumar Karampuri wrote:
Okay good. At least this validates my doubts. Handling O_SYNC in 
gluster NFS and fuse is a bit different.
When application opens a file with O_SYNC on fuse mount then each 
write syscall has to be written to disk as part of the syscall where 
as in case of NFS, there is no concept of open. NFS performs write 
though a handle saying it needs to be a synchronous write, so write() 
syscall is performed first then it performs fsync(). so an write on an 
fd with O_SYNC becomes write+fsync. I am suspecting that when multiple 
threads do this write+fsync() operation on the same file, multiple 
writes are batched together to be written do disk so the throughput on 
the disk is increasing is my guess.


Does it answer your doubts?

On Wed, May 10, 2017 at 9:35 PM, Pat Haley > wrote:



Without the oflag=sync and only a single test of each, the FUSE is
going faster than NFS:

FUSE:
mseas-data2(dri_nascar)% dd if=/dev/zero count=4096 bs=1048576
of=zeros.txt conv=sync
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 7.46961 s, 575 MB/s


NFS
mseas-data2(HYCOM)% dd if=/dev/zero count=4096 bs=1048576
of=zeros.txt conv=sync
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 11.4264 s, 376 MB/s



On 05/10/2017 11:53 AM, Pranith Kumar Karampuri wrote:

Could you let me know the speed without oflag=sync on both the
mounts? No need to collect profiles.

On Wed, May 10, 2017 at 9:17 PM, Pat Haley > wrote:


Here is what I see now:

[root@mseas-data2 ~]# gluster volume info

Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off



On 05/10/2017 11:44 AM, Pranith Kumar Karampuri wrote:

Is this the volume info you have?

>/[root at mseas-data2
 ~]#
gluster volume info />//>/Volume Name: data-volume />/Type: Distribute />/Volume ID: 
c162161e-2a2d-4dac-b015-f31fd89ceb18 />/Status: Started />/Number of Bricks: 2 />/Transport-type: tcp 
/>/Bricks: />/Brick1: mseas-data2:/mnt/brick1 />/Brick2: mseas-data2:/mnt/brick2 />/Options Reconfigured: 
/>/performance.readdir-ahead: on />/nfs.disable: on />/nfs.export-volumes: off /
​I copied this from old thread from 2016. This is distribute
volume. Did you change any of the options in between?


-- 


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:pha...@mit.edu 

Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

-- 
Pranith
-- 


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:pha...@mit.edu 

Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

--
Pranith

--

Re: [Gluster-users] Slow write times to gluster disk

[Pranith Kumar Karampuri]

Okay good. At least this validates my doubts. Handling O_SYNC in gluster
NFS and fuse is a bit different.
When application opens a file with O_SYNC on fuse mount then each write
syscall has to be written to disk as part of the syscall where as in case
of NFS, there is no concept of open. NFS performs write though a handle
saying it needs to be a synchronous write, so write() syscall is performed
first then it performs fsync(). so an write on an fd with O_SYNC becomes
write+fsync. I am suspecting that when multiple threads do this
write+fsync() operation on the same file, multiple writes are batched
together to be written do disk so the throughput on the disk is increasing
is my guess.

Does it answer your doubts?

On Wed, May 10, 2017 at 9:35 PM, Pat Haley  wrote:

>
> Without the oflag=sync and only a single test of each, the FUSE is going
> faster than NFS:
>
> FUSE:
> mseas-data2(dri_nascar)% dd if=/dev/zero count=4096 bs=1048576
> of=zeros.txt conv=sync
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes (4.3 GB) copied, 7.46961 s, 575 MB/s
>
>
> NFS
> mseas-data2(HYCOM)% dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt
> conv=sync
> 4096+0 records in
> 4096+0 records out
> 4294967296 bytes (4.3 GB) copied, 11.4264 s, 376 MB/s
>
>
>
> On 05/10/2017 11:53 AM, Pranith Kumar Karampuri wrote:
>
> Could you let me know the speed without oflag=sync on both the mounts? No
> need to collect profiles.
>
> On Wed, May 10, 2017 at 9:17 PM, Pat Haley  wrote:
>
>>
>> Here is what I see now:
>>
>> [root@mseas-data2 ~]# gluster volume info
>>
>> Volume Name: data-volume
>> Type: Distribute
>> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>> Status: Started
>> Number of Bricks: 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: mseas-data2:/mnt/brick1
>> Brick2: mseas-data2:/mnt/brick2
>> Options Reconfigured:
>> diagnostics.count-fop-hits: on
>> diagnostics.latency-measurement: on
>> nfs.exports-auth-enable: on
>> diagnostics.brick-sys-log-level: WARNING
>> performance.readdir-ahead: on
>> nfs.disable: on
>> nfs.export-volumes: off
>>
>>
>>
>> On 05/10/2017 11:44 AM, Pranith Kumar Karampuri wrote:
>>
>> Is this the volume info you have?
>>
>> >* [root at mseas-data2 
>> > ~]# gluster volume 
>> >info
>> *>>* Volume Name: data-volume
>> *>* Type: Distribute
>> *>* Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>> *>* Status: Started
>> *>* Number of Bricks: 2
>> *>* Transport-type: tcp
>> *>* Bricks:
>> *>* Brick1: mseas-data2:/mnt/brick1
>> *>* Brick2: mseas-data2:/mnt/brick2
>> *>* Options Reconfigured:
>> *>* performance.readdir-ahead: on
>> *>* nfs.disable: on
>> *>* nfs.export-volumes: off
>>
>> *
>>
>> ​I copied this from old thread from 2016. This is distribute volume. Did
>> you change any of the options in between?
>>
>> --
>>
>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>> Pat Haley  Email:  pha...@mit.edu
>> Center for Ocean Engineering   Phone:  (617) 253-6824
>> Dept. of Mechanical EngineeringFax:(617) 253-8125
>> MIT, Room 5-213http://web.mit.edu/phaley/www/
>> 77 Massachusetts Avenue
>> Cambridge, MA  02139-4301
>>
>> --
> Pranith
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley  Email:  pha...@mit.edu
> Center for Ocean Engineering   Phone:  (617) 253-6824
> Dept. of Mechanical EngineeringFax:(617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
>
>


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow write times to gluster disk

[Pat Haley]

Without the oflag=sync and only a single test of each, the FUSE is going 
faster than NFS:


FUSE:
mseas-data2(dri_nascar)% dd if=/dev/zero count=4096 bs=1048576 
of=zeros.txt conv=sync

4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 7.46961 s, 575 MB/s


NFS
mseas-data2(HYCOM)% dd if=/dev/zero count=4096 bs=1048576 of=zeros.txt 
conv=sync

4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 11.4264 s, 376 MB/s


On 05/10/2017 11:53 AM, Pranith Kumar Karampuri wrote:
Could you let me know the speed without oflag=sync on both the mounts? 
No need to collect profiles.


On Wed, May 10, 2017 at 9:17 PM, Pat Haley > wrote:



Here is what I see now:

[root@mseas-data2 ~]# gluster volume info

Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off



On 05/10/2017 11:44 AM, Pranith Kumar Karampuri wrote:

Is this the volume info you have?

>/[root at mseas-data2
 ~]#
gluster volume info />//>/Volume Name: data-volume />/Type: Distribute />/Volume ID: 
c162161e-2a2d-4dac-b015-f31fd89ceb18 />/Status: Started />/Number of Bricks: 2 />/Transport-type: tcp 
/>/Bricks: />/Brick1: mseas-data2:/mnt/brick1 />/Brick2: mseas-data2:/mnt/brick2 />/Options Reconfigured: 
/>/performance.readdir-ahead: on />/nfs.disable: on />/nfs.export-volumes: off /
​I copied this from old thread from 2016. This is distribute
volume. Did you change any of the options in between?


-- 


-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:pha...@mit.edu 

Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

--
Pranith

--

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:  pha...@mit.edu
Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow write times to gluster disk

[Pranith Kumar Karampuri]

Could you let me know the speed without oflag=sync on both the mounts? No
need to collect profiles.

On Wed, May 10, 2017 at 9:17 PM, Pat Haley  wrote:

>
> Here is what I see now:
>
> [root@mseas-data2 ~]# gluster volume info
>
> Volume Name: data-volume
> Type: Distribute
> Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
> Status: Started
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: mseas-data2:/mnt/brick1
> Brick2: mseas-data2:/mnt/brick2
> Options Reconfigured:
> diagnostics.count-fop-hits: on
> diagnostics.latency-measurement: on
> nfs.exports-auth-enable: on
> diagnostics.brick-sys-log-level: WARNING
> performance.readdir-ahead: on
> nfs.disable: on
> nfs.export-volumes: off
>
>
>
> On 05/10/2017 11:44 AM, Pranith Kumar Karampuri wrote:
>
> Is this the volume info you have?
>
> >* [root at mseas-data2 
> > ~]# gluster volume 
> >info
> *>>* Volume Name: data-volume
> *>* Type: Distribute
> *>* Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
> *>* Status: Started
> *>* Number of Bricks: 2
> *>* Transport-type: tcp
> *>* Bricks:
> *>* Brick1: mseas-data2:/mnt/brick1
> *>* Brick2: mseas-data2:/mnt/brick2
> *>* Options Reconfigured:
> *>* performance.readdir-ahead: on
> *>* nfs.disable: on
> *>* nfs.export-volumes: off
>
> *
>
> ​I copied this from old thread from 2016. This is distribute volume. Did
> you change any of the options in between?
>
>
> --
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Pat Haley  Email:  pha...@mit.edu
> Center for Ocean Engineering   Phone:  (617) 253-6824
> Dept. of Mechanical EngineeringFax:(617) 253-8125
> MIT, Room 5-213http://web.mit.edu/phaley/www/
> 77 Massachusetts Avenue
> Cambridge, MA  02139-4301
>
>


-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow write times to gluster disk

[Pat Haley]

Here is what I see now:

[root@mseas-data2 ~]# gluster volume info

Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off



On 05/10/2017 11:44 AM, Pranith Kumar Karampuri wrote:

Is this the volume info you have?

>/[root at mseas-data2 
 ~]# gluster 
volume info />//>/Volume Name: data-volume />/Type: Distribute />/Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18 />/Status: Started />/Number of Bricks: 2 />/Transport-type: tcp />/Bricks: />/Brick1: mseas-data2:/mnt/brick1 />/Brick2: mseas-data2:/mnt/brick2 />/Options Reconfigured: />/performance.readdir-ahead: on />/nfs.disable: on />/nfs.export-volumes: off /
​I copied this from old thread from 2016. This is distribute volume. 
Did you change any of the options in between?


--

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley  Email:  pha...@mit.edu
Center for Ocean Engineering   Phone:  (617) 253-6824
Dept. of Mechanical EngineeringFax:(617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow write times to gluster disk

[Pranith Kumar Karampuri]

Is this the volume info you have?

>* [root at mseas-data2 
> ~]# gluster volume info
*>>* Volume Name: data-volume
*>* Type: Distribute
*>* Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
*>* Status: Started
*>* Number of Bricks: 2
*>* Transport-type: tcp
*>* Bricks:
*>* Brick1: mseas-data2:/mnt/brick1
*>* Brick2: mseas-data2:/mnt/brick2
*>* Options Reconfigured:
*>* performance.readdir-ahead: on
*>* nfs.disable: on
*>

* nfs.export-volumes: off*

​I copied this from old thread from 2016. This is distribute volume. Did
you change any of the options in between?
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Slow write times to gluster disk

[Pat Haley]

Hi,

We finally managed to do the dd tests for an NFS-mounted gluster file 
system.  The profile results during that test are in


http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_nfs_test

The summary of the dd tests are

 * writing to gluster disk mounted with fuse:  5 Mb/s
 * writing to gluster disk mounted with nfs:  200 Mb/s

Pat


On 05/05/2017 08:11 PM, Pat Haley wrote:


Hi,

We redid the dd tests (this time using conv=sync oflag=sync to avoid 
caching questions).  The profile results are in


http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_fuse_test


On 05/05/2017 12:47 PM, Ravishankar N wrote:

On 05/05/2017 08:42 PM, Pat Haley wrote:


Hi Pranith,

I presume you are asking for some version of the profile data that 
just shows the dd test (or a repeat of the dd test).  If yes, how do 
I extract just that data?
Yes, that is what he is asking for. Just clear the existing profile 
info using `gluster volume profile volname clear` and run the dd test 
once. Then when you run profile info again, it should just give you 
the stats for the dd test.


Thanks

Pat



On 05/05/2017 10:58 AM, Pranith Kumar Karampuri wrote:

hi Pat,
  Let us concentrate on the performance numbers part for now. 
We will look at the permissions one after this?


As per the profile info, only 2.6% of the work-load is writes. 
There are too many Lookups.


Would it be possible to get the data for just the dd test you were 
doing earlier?



On Fri, May 5, 2017 at 8:14 PM, Pat Haley > wrote:



Hi Pranith & Ravi,

A couple of quick questions

We have profile turned on. Are there specific queries we should
make that would help debug our configuration?  (The default
profile info was previously sent in
http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html

but I'm not sure if that is what you were looking for.)

We also started to do a test on serving gluster over NFS.  We
rediscovered an issue we previously reported (
http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html


) in that the NFS mounted version was ignoring the group write
permissions.  What specific information would be useful in
debugging this?

Thanks

Pat



On 04/14/2017 03:01 AM, Ravishankar N wrote:

On 04/14/2017 12:20 PM, Pranith Kumar Karampuri wrote:



On Sat, Apr 8, 2017 at 10:28 AM, Ravishankar N
> wrote:

Hi Pat,

I'm assuming you are using gluster native (fuse mount).
If it helps, you could try mounting it via gluster NFS
(gnfs) and then see if there is an improvement in speed.
Fuse mounts are slower than gnfs mounts but you get the
benefit of avoiding a single point of failure. Unlike
fuse mounts, if the gluster node containing the gnfs
server goes down, all mounts done using that node will
fail). For fuse mounts, you could try tweaking the
write-behind xlator settings to see if it helps. See the
performance.write-behind and
performance.write-behind-window-size options in `gluster
volume set help`. Of course, even for gnfs mounts, you
can achieve fail-over by using CTDB.


Ravi,
  Do you have any data that suggests fuse mounts are
slower than gNFS servers?

I have heard anecdotal evidence time and again on the ML and
IRC, which is why I wanted to compare it with NFS numbers on
his setup.


Pat,
  I see that I am late to the thread, but do you happen
to have "profile info" of the workload?

You can follow

https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/


to get the information.

Yeah, Let's see if profile info shows up anything interesting.
-Ravi



Thanks,
Ravi


On 04/08/2017 12:07 AM, Pat Haley wrote:


Hi,

We noticed a dramatic slowness when writing to a gluster
disk when compared to writing to an NFS disk.
Specifically when using dd (data duplicator) to write a
4.3 GB file of zeros:

  * on NFS disk (/home): 9.5 Gb/s
  * on gluster disk (/gdata): 508 Mb/s

The gluser disk is 2 bricks joined together, no
replication or anything else. The hardware is
(literally) the same:

  * one server with 70 hard disks  and a hardware RAID card.
  * 4 disks in a RAID-6 group (the NFS disk)
  * 32 disks in a RAID-6 group (the max allowed by the
card, /mnt/brick1)
  * 32 disks in another RAID-6 group (/mnt/brick2)
  

Re: [Gluster-users] VM going down

[Niels de Vos]

On Wed, May 10, 2017 at 04:08:22PM +0530, Pranith Kumar Karampuri wrote:
> On Tue, May 9, 2017 at 7:40 PM, Niels de Vos  wrote:
> 
> > ...
> > > > client from
> > > > srvpve2-162483-2017/05/08-10:01:06:189720-datastore2-client-0-0-0
> > > > (version: 3.8.11)
> > > > [2017-05-08 10:01:06.237433] E [MSGID: 113107]
> > [posix.c:1079:posix_seek]
> > > > 0-datastore2-posix: seek failed on fd 18 length 42957209600 [No such
> > > > device or address]
> >
> > The SEEK procedure translates to lseek() in the posix xlator. This can
> > return with "No suck device or address" (ENXIO) in only one case:
> >
> > ENXIOwhence is SEEK_DATA or SEEK_HOLE, and the file offset is
> >  beyond the end of the file.
> >
> > This means that an lseek() was executed where the current offset of the
> > filedescriptor was higher than the size of the file. I'm not sure how
> > that could happen... Sharding prevents using SEEK at all atm.
> >
> > ...
> > > > The strange part is that I cannot seem to find any other error.
> > > > If I restart the VM everything works as expected (it stopped at ~9.51
> > > > UTC and was started at ~10.01 UTC) .
> > > >
> > > > This is not the first time that this happened, and I do not see any
> > > > problems with networking or the hosts.
> > > >
> > > > Gluster version is 3.8.11
> > > > this is the incriminated volume (though it happened on a different one
> > too)
> > > >
> > > > Volume Name: datastore2
> > > > Type: Replicate
> > > > Volume ID: c95ebb5f-6e04-4f09-91b9-bbbe63d83aea
> > > > Status: Started
> > > > Snapshot Count: 0
> > > > Number of Bricks: 1 x (2 + 1) = 3
> > > > Transport-type: tcp
> > > > Bricks:
> > > > Brick1: srvpve2g:/data/brick2/brick
> > > > Brick2: srvpve3g:/data/brick2/brick
> > > > Brick3: srvpve1g:/data/brick2/brick (arbiter)
> > > > Options Reconfigured:
> > > > nfs.disable: on
> > > > performance.readdir-ahead: on
> > > > transport.address-family: inet
> > > >
> > > > Any hint on how to dig more deeply into the reason would be greatly
> > > > appreciated.
> >
> > Probably the problem is with SEEK support in the arbiter functionality.
> > Just like with a READ or a WRITE on the arbiter brick, SEEK can only
> > succeed on bricks where the files with content are located. It does not
> > look like arbiter handles SEEK, so the offset in lseek() will likely be
> > higher than the size of the file on the brick (empty, 0 size file). I
> > don't know how the replication xlator responds on an error return from
> > SEEK on one of the bricks, but I doubt it likes it.
> >
> 
> inode-read fops don't get sent to arbiter brick. So this won't happen.

Yes, I see that the arbiter xlator returns on reads without going to the
bricks. Should that not be done for seek as well? It's the first time I
actually looked at the code of the arbiter xlator, so I might well be
misunderstanding how it works :)

Thanks,
Niels


> 
> 
> >
> > We have https://bugzilla.redhat.com/show_bug.cgi?id=1301647 to support
> > SEEK for sharding. I suggest you open a bug for getting SEEK in the
> > arbiter xlator as well.
> >
> > HTH,
> > Niels
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> >
> 
> 
> 
> -- 
> Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] small files optimizations

[Pranith Kumar Karampuri]

I think you should take a look at:
https://wiki.dovecot.org/MailLocation/SharedDisk


if not already.

On Wed, May 10, 2017 at 12:52 PM,  wrote:

> On Wed, May 10, 2017 at 09:14:59AM +0200, Gandalf Corvotempesta wrote:
> > Yes much clearer but I think this makes some trouble like space available
> > shown by gluster. Or not?
>
> Not really, you'll just see "used space" on your volumes that you won't be
> able to track down, keep in mind that the used space on each volume will be
> the same (gluster uses the used space from the brick's filesystem)
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] VM going down

[Pranith Kumar Karampuri]

On Tue, May 9, 2017 at 7:40 PM, Niels de Vos  wrote:

> ...
> > > client from
> > > srvpve2-162483-2017/05/08-10:01:06:189720-datastore2-client-0-0-0
> > > (version: 3.8.11)
> > > [2017-05-08 10:01:06.237433] E [MSGID: 113107]
> [posix.c:1079:posix_seek]
> > > 0-datastore2-posix: seek failed on fd 18 length 42957209600 [No such
> > > device or address]
>
> The SEEK procedure translates to lseek() in the posix xlator. This can
> return with "No suck device or address" (ENXIO) in only one case:
>
> ENXIOwhence is SEEK_DATA or SEEK_HOLE, and the file offset is
>  beyond the end of the file.
>
> This means that an lseek() was executed where the current offset of the
> filedescriptor was higher than the size of the file. I'm not sure how
> that could happen... Sharding prevents using SEEK at all atm.
>
> ...
> > > The strange part is that I cannot seem to find any other error.
> > > If I restart the VM everything works as expected (it stopped at ~9.51
> > > UTC and was started at ~10.01 UTC) .
> > >
> > > This is not the first time that this happened, and I do not see any
> > > problems with networking or the hosts.
> > >
> > > Gluster version is 3.8.11
> > > this is the incriminated volume (though it happened on a different one
> too)
> > >
> > > Volume Name: datastore2
> > > Type: Replicate
> > > Volume ID: c95ebb5f-6e04-4f09-91b9-bbbe63d83aea
> > > Status: Started
> > > Snapshot Count: 0
> > > Number of Bricks: 1 x (2 + 1) = 3
> > > Transport-type: tcp
> > > Bricks:
> > > Brick1: srvpve2g:/data/brick2/brick
> > > Brick2: srvpve3g:/data/brick2/brick
> > > Brick3: srvpve1g:/data/brick2/brick (arbiter)
> > > Options Reconfigured:
> > > nfs.disable: on
> > > performance.readdir-ahead: on
> > > transport.address-family: inet
> > >
> > > Any hint on how to dig more deeply into the reason would be greatly
> > > appreciated.
>
> Probably the problem is with SEEK support in the arbiter functionality.
> Just like with a READ or a WRITE on the arbiter brick, SEEK can only
> succeed on bricks where the files with content are located. It does not
> look like arbiter handles SEEK, so the offset in lseek() will likely be
> higher than the size of the file on the brick (empty, 0 size file). I
> don't know how the replication xlator responds on an error return from
> SEEK on one of the bricks, but I doubt it likes it.
>

inode-read fops don't get sent to arbiter brick. So this won't happen.


>
> We have https://bugzilla.redhat.com/show_bug.cgi?id=1301647 to support
> SEEK for sharding. I suggest you open a bug for getting SEEK in the
> arbiter xlator as well.
>
> HTH,
> Niels
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gdeploy not starting all the daemons for NFS-ganesha :(

[Manisha Saini]

Hi hvjunk

>From the logs it seems like the ganesha cluster is not yet created/running and 
>performing refresh config is failing.As the cluster is not running and volume 
>"ganesha" is not yet exported due to this performing refresh config on volume 
>will fail.
For creating ganesha cluster,there are few pre-requisite of adding services 
like  nlm,nfs,rpc-bind,high-availability,mountd,rquota to firewalld.
After executing create-cluster block ,It will validate the HA Status of ganesha 
via gdeploy.If the ganesha cluster is up and running it will reflect HA status 
as "HEALTHY".
After then performing refresh-config on nfs exported volume will succeed.


Attaching the sample config file to create ganesha cluster and exporting 
ganesha volume to client -

[hosts]
dhcp37-102.lab.eng.blr.redhat.com
dhcp37-92.lab.eng.blr.redhat.com
dhcp37-119.lab.eng.blr.redhat.com
dhcp37-122.lab.eng.blr.redhat.com


[backend-setup]
devices=/dev/sdb,/dev/sdc,/dev/sdd
vgs=vg1,vg2,vg3
pools=pool1,pool2,pool3
lvs=lv1,lv2,lv3
mountpoints=/gluster/brick1,/gluster/brick2,/gluster/brick3
brick_dirs=/gluster/brick1/1,/gluster/brick2/1,/gluster/brick3/1

[yum]
action=install
repolist=
gpgcheck=no
update=no
packages=glusterfs-ganesha

[firewalld]
action=add
ports=111/tcp,2049/tcp,54321/tcp,5900/tcp,5900-6923/tcp,5666/tcp,16514/tcp
services=glusterfs,nlm,nfs,rpc-bind,high-availability,mountd,rquota

[volume]
action=create
volname=ganesha
transport=tcp
replica_count=2
force=yes

[nfs-ganesha]
action=create-cluster
ha-name=ganesha-ha-360
cluster-nodes=dhcp37-102.lab.eng.blr.redhat.com,dhcp37-92.lab.eng.blr.redhat.com,dhcp37-119.lab.eng.blr.redhat.com,dhcp37-122.lab.eng.blr.redhat.com
vip=10.70.36.217,10.70.36.218,10.70.36.219,10.70.36.220
volname=ganesha  #This will export volume via nfs-ganesha

[clients]
action=mount
volname=10.70.37.153:/ganesha  #VIP from which the volume needs to be mounted 
to client
hosts=10.70.46.30#Client IP
client_mount_points=/mnt/ganesha
fstype=nfs
options=rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.70.44.153,local_lock=none,addr=10.70.46.30

Regards,
Manisha Saini

- Original Message -
From: "Soumya Koduri" 
To: "hvjunk" , gluster-users@gluster.org, "Sachidananda URS" 
, "Manisha Saini" , "Arthy Loganathan" 

Sent: Wednesday, May 10, 2017 10:45:57 AM
Subject: Re: [Gluster-users] gdeploy not starting all the daemons for 
NFS-ganesha :(

CCin Sac, Manisha, Arthy who could help with troubleshooting.

Thanks,
Soumya


On 05/09/2017 08:31 PM, hvjunk wrote:
> Hi there,
>
> Given the following config file, what am I doing wrong causing the error at 
> the bottom  & no mounted /gluster_shared_storage?
>
> Hendrik
>
> [root@linked-clone-of-centos-linux ~]# cat t.conf
> [hosts]
> 10.10.10.11
> 10.10.10.12
> 10.10.10.13
>
> [backend-setup]
> devices=/dev/sdb
> mountpoints=/gluster/brick1
> brick_dirs=/gluster/brick1/one
> pools=pool1
>
> #Installing nfs-ganesha
> [yum]
> action=install
> repolist=
> gpgcheck=no
> update=no
> packages=glusterfs-ganesha
>
> [service]
> action=start
> service=glusterd
>
> [service]
> action=enable
> service=glusterd
>
> #This will create a volume. Skip this section if your volume already exists
> [volume]
> action=create
> volname=ganesha
> transport=tcp
> replica_count=3
> arbiter_count=1
> force=yes
>
> [clients]
> action=mount
> volname=ganesha
> hosts=10.10.10.11,10.10.10.12,10.10.10.13
> fstype=glusterfs
> client_mount_points=/mnt/ganesha_mnt
>
>
> #Creating a high availability cluster and exporting the volume
> [nfs-ganesha]
> action=create-cluster
> ha-name=ganesha-ha-360
> cluster-nodes=10.10.10.11,10.10.10.12
> vip=10.10.10.31,10.10.10.41
> volname=ganesha
>
>
> [nfs-ganesha]
> action=export-volume
> volname=ganesha
>
> [nfs-ganesha]
> action=refresh-config
> volname=ganesha
> [root@linked-clone-of-centos-linux ~]#
>
>
> [root@linked-clone-of-centos-linux ~]# gdeploy -c t.conf -k
>
> PLAY [gluster_servers] 
> *
>
> TASK [Clean up filesystem signature] 
> ***
> skipping: [10.10.10.11] => (item=/dev/sdb)
> skipping: [10.10.10.12] => (item=/dev/sdb)
> skipping: [10.10.10.13] => (item=/dev/sdb)
>
> TASK [Create Physical Volume] 
> **
> changed: [10.10.10.13] => (item=/dev/sdb)
> changed: [10.10.10.11] => (item=/dev/sdb)
> changed: [10.10.10.12] => (item=/dev/sdb)
>
> PLAY RECAP 
> *
> 10.10.10.11: ok=1changed=1unreachable=0failed=0
> 10.10.10.12: ok=1changed=1unreachable=0failed=0
> 10.10.10.13: ok=1changed=1   

[Gluster-users] GlusterFS+heketi+Kubernetes snapshots fail

[Chris Jones]

Hi All,

This was discussed briefly on IRC, but got no resolution. I have a 
Kubernetes cluster running heketi and GlusterFS 3.10.1. When I try to 
create a snapshot, I get:


snapshot create: failed: Commit failed on localhost. Please check log 
file for details.


glusterd log: http://termbin.com/r8s3

brick log: http://termbin.com/l0ya

lvs output: http://termbin.com/bwug

"gluster snapshot config" output: http://termbin.com/4t1k

As you can see, there's not a lot of helpful output in the log files. 
I'd be grateful if somebody could help me interpret what's there.


Chris

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Poor performance for nfs client on windows than on linux

[ChiKu]

Hello,

I'm testing glusterfs for windows client.
I created 2 servers for glusterfs (3.10.1 replication 2) on centos 7.3.

Right now, I just use default setting and my testing use case is alot small
files in a folder.

nfs windows client is so poor performance than nfs linux client. Idon't
understand. It should have same nfs linux performance.
I saw something wierd about network traffic. On windows client I saw more
receive (9Mbps) traffic than send traffic (1Mpbs).

On nfs linux client, receive traffic is around 700Kbps.

Can someone have any idea what happen with nfs windows client?
I will try later some tunning tests.




* 1st test: centos client mount with glusterfs type :
gl1.lab.com:vol1 on /mnt/glusterfs type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)


python smallfile_cli.py --operation create --threads 1 --file-size 30
--files 5000 --files-per-dir 1 --top /mnt/glusterfs/test1
smallfile version 3.0
   hosts in test : None
   top test directory(s) : ['/mnt/glusterfs/test1']
   operation : create
files/thread : 5000
 threads : 1
   record size (KB, 0 = maximum) : 0
  file size (KB) : 30
  file size distribution : fixed
   files per dir : 1
dirs per dir : 10
  threads share directories? : N
 filename prefix :
 filename suffix :
 hash file number into dir.? : N
 fsync after modify? : N
  pause between files (microsec) : 0
finish all requests? : Y
  stonewall? : Y
 measure response times? : N
verify read? : Y
verbose? : False
  log to stderr? : False
   ext.attr.size : 0
  ext.attr.count : 0
host = cm2.lab.com,thr = 00,elapsed = 16.566169,files = 5000,records =
5000,status = ok
total threads = 1
total files = 5000
total data = 0.143 GB
100.00% of requested files processed, minimum is  90.00
16.566169 sec elapsed time
301.819932 files/sec
301.819932 IOPS
8.842381 MB/sec

* 2nd test centos client mount with nfs :
gl1.lab.com:/vol1 on /mnt/nfs type nfs
(rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.47.11,mountvers=3,mountport=38465,mountproto=tcp,local_lock=none,addr=192.168.47.11)


python smallfile_cli.py --operation create --threads 1 --file-size 30
--files 5000 --files-per-dir 1 --top /mnt/nfs/test1
smallfile version 3.0
   hosts in test : None
   top test directory(s) : ['/mnt/nfs/test1']
   operation : create
files/thread : 5000
 threads : 1
   record size (KB, 0 = maximum) : 0
  file size (KB) : 30
  file size distribution : fixed
   files per dir : 1
dirs per dir : 10
  threads share directories? : N
 filename prefix :
 filename suffix :
 hash file number into dir.? : N
 fsync after modify? : N
  pause between files (microsec) : 0
finish all requests? : Y
  stonewall? : Y
 measure response times? : N
verify read? : Y
verbose? : False
  log to stderr? : False
   ext.attr.size : 0
  ext.attr.count : 0
host = cm2.lab.com,thr = 00,elapsed = 54.737751,files = 5000,records =
5000,status = ok
total threads = 1
total files = 5000
total data = 0.143 GB
100.00% of requested files processed, minimum is  90.00
54.737751 sec elapsed time
91.344637 files/sec
91.344637 IOPS
2.676112 MB/sec


* 3th test: new windows 2012R2 with nfs client installed :

C:\Users\Administrator\smallfile>smallfile_cli.py --operation create
--threads 1 --file-size 30 --files 5000 --files-per-dir 1 --top
\\192.168.47.11\vol1\test1
smallfile version 3.0
   hosts in test : None
   top test directory(s) :
['192.168.47.11\\vol1\\test1']
   operation : create
files/thread : 5000
 threads : 1
   record size (KB, 0 = maximum) : 0
  file size (KB) : 30
  file size distribution : fixed
   files per dir : 1
dirs per dir : 10
   

[Gluster-users] Community meeting 2017-05-10

[Kaushal M]

Hi all,

Today's meeting is scheduled to happen in 6 hours at 1500UTC. The
meeting pad is at https://bit.ly/gluster-community-meetings . Please
add your updates and topics for discussion.

I had forgotten to send out the meeting minutes and logs for the last
meeting which happened on 2017-04-26. There wasn't a lot of discussion
in the meeting, but the logs and minutes are available at the links
below.

Minutes: 
https://meetbot.fedoraproject.org/gluster-meeting/2017-04-26/gluster_community_meeting_2017-04-26.2017-04-26-15.00.html
Minutes (text):
https://meetbot.fedoraproject.org/gluster-meeting/2017-04-26/gluster_community_meeting_2017-04-26.2017-04-26-15.00.txt
Log: 
https://meetbot.fedoraproject.org/gluster-meeting/2017-04-26/gluster_community_meeting_2017-04-26.2017-04-26-15.00.log.html

~kaushal
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] plan to use as a nas for homedirs

[Laurent Bardi]

Hi,

I have 2x (40TB DAS attached  to a machine running ESX (64Go RAM)). I 
plan to use them as an active-active cluster with gluster + ctdb 
(serving only smb no nfs). The goal is to store homedirs and profiles  
for all my client machines (~ 150 win7 client).


Any advice are welcome !

-Should i use only one VM with gluster + ctdb on each esx , each VM 
accessing a raw disk of 40TB


or

-Should i use 4 or 8 VM on each esx with 4x10TB or 8x5TB raw disks (each 
VM driving a single rawdisk)


I there anybody on the list who already achieve that ?

Many thanks in advance.


--
Laurent BARDI /  RSI CNRS-IPBS / CRSSI DR14
INSTITUT  de PHARMACOLOGIE et de BIOLOGIE STRUCTURALE
Tel : 05-61-17-59-05http://www.ipbs.fr/
GSM : 06-23-46-06-28Laurent.BardiATipbs.fr
CNRS-IPBS 205 Route de Narbonne 31400 TOULOUSE FRANCE
...
J'étais indéniablement misanthrope.
Je voulus traverser à gué un marigot infesté d'imbéciles.
Quand j'atteignis l'autre rive, j'étais devenu philanthrope.

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] gdeploy not starting all the daemons for NFS-ganesha :(

[Manisha Saini]

Hi hvjunk

>From the logs it seems like the ganesha cluster is not yet created/running and 
>performing refresh config is failing.As the cluster is not running and volume 
>"ganesha" is not yet exported due to this performing refresh config on volume 
>will fail.
For creating ganesha cluster,there are few pre-requisite of adding services 
like  nlm,nfs,rpc-bind,high-availability,mountd,rquota to firewalld.
After executing create-cluster block ,It will validate the HA Status of ganesha 
via gdeploy.If the ganesha cluster is up and running it will reflect HA status 
as "HEALTHY".
After then performing refresh-config on nfs exported volume will succeed.


Attaching the sample config file to create ganesha cluster and exporting 
ganesha volume to client -

[hosts]
dhcp37-102.lab.eng.blr.redhat.com
dhcp37-92.lab.eng.blr.redhat.com
dhcp37-119.lab.eng.blr.redhat.com
dhcp37-122.lab.eng.blr.redhat.com


[backend-setup]
devices=/dev/sdb,/dev/sdc,/dev/sdd
vgs=vg1,vg2,vg3
pools=pool1,pool2,pool3
lvs=lv1,lv2,lv3
mountpoints=/gluster/brick1,/gluster/brick2,/gluster/brick3
brick_dirs=/gluster/brick1/1,/gluster/brick2/1,/gluster/brick3/1

[yum]
action=install
repolist=
gpgcheck=no
update=no
packages=glusterfs-ganesha

[firewalld]
action=add
ports=111/tcp,2049/tcp,54321/tcp,5900/tcp,5900-6923/tcp,5666/tcp,16514/tcp
services=glusterfs,nlm,nfs,rpc-bind,high-availability,mountd,rquota

[volume]
action=create
volname=ganesha
transport=tcp
replica_count=2
force=yes

[nfs-ganesha]
action=create-cluster
ha-name=ganesha-ha-360
cluster-nodes=dhcp37-102.lab.eng.blr.redhat.com,dhcp37-92.lab.eng.blr.redhat.com,dhcp37-119.lab.eng.blr.redhat.com,dhcp37-122.lab.eng.blr.redhat.com
vip=10.70.36.217,10.70.36.218,10.70.36.219,10.70.36.220
volname=ganesha  #This will export volume via nfs-ganesha

[clients]
action=mount
volname=10.70.37.153:/ganesha  #VIP from which the volume needs to be mounted 
to client
hosts=10.70.46.30#Client IP
client_mount_points=/mnt/ganesha
fstype=nfs
options=rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.70.44.153,local_lock=none,addr=10.70.46.30

Regards,
Manisha Saini

- Original Message -
From: "Soumya Koduri" 
To: "hvjunk" , gluster-users@gluster.org, "Sachidananda URS" 
, "Manisha Saini" , "Arthy Loganathan" 

Sent: Wednesday, May 10, 2017 10:45:57 AM
Subject: Re: [Gluster-users] gdeploy not starting all the daemons for 
NFS-ganesha :(

CCin Sac, Manisha, Arthy who could help with troubleshooting.

Thanks,
Soumya


On 05/09/2017 08:31 PM, hvjunk wrote:
> Hi there,
>
> Given the following config file, what am I doing wrong causing the error at 
> the bottom  & no mounted /gluster_shared_storage?
>
> Hendrik
>
> [root@linked-clone-of-centos-linux ~]# cat t.conf
> [hosts]
> 10.10.10.11
> 10.10.10.12
> 10.10.10.13
>
> [backend-setup]
> devices=/dev/sdb
> mountpoints=/gluster/brick1
> brick_dirs=/gluster/brick1/one
> pools=pool1
>
> #Installing nfs-ganesha
> [yum]
> action=install
> repolist=
> gpgcheck=no
> update=no
> packages=glusterfs-ganesha
>
> [service]
> action=start
> service=glusterd
>
> [service]
> action=enable
> service=glusterd
>
> #This will create a volume. Skip this section if your volume already exists
> [volume]
> action=create
> volname=ganesha
> transport=tcp
> replica_count=3
> arbiter_count=1
> force=yes
>
> [clients]
> action=mount
> volname=ganesha
> hosts=10.10.10.11,10.10.10.12,10.10.10.13
> fstype=glusterfs
> client_mount_points=/mnt/ganesha_mnt
>
>
> #Creating a high availability cluster and exporting the volume
> [nfs-ganesha]
> action=create-cluster
> ha-name=ganesha-ha-360
> cluster-nodes=10.10.10.11,10.10.10.12
> vip=10.10.10.31,10.10.10.41
> volname=ganesha
>
>
> [nfs-ganesha]
> action=export-volume
> volname=ganesha
>
> [nfs-ganesha]
> action=refresh-config
> volname=ganesha
> [root@linked-clone-of-centos-linux ~]#
>
>
> [root@linked-clone-of-centos-linux ~]# gdeploy -c t.conf -k
>
> PLAY [gluster_servers] 
> *
>
> TASK [Clean up filesystem signature] 
> ***
> skipping: [10.10.10.11] => (item=/dev/sdb)
> skipping: [10.10.10.12] => (item=/dev/sdb)
> skipping: [10.10.10.13] => (item=/dev/sdb)
>
> TASK [Create Physical Volume] 
> **
> changed: [10.10.10.13] => (item=/dev/sdb)
> changed: [10.10.10.11] => (item=/dev/sdb)
> changed: [10.10.10.12] => (item=/dev/sdb)
>
> PLAY RECAP 
> *
> 10.10.10.11: ok=1changed=1unreachable=0failed=0
> 10.10.10.12: ok=1changed=1unreachable=0failed=0
> 10.10.10.13: ok=1changed=1   

Re: [Gluster-users] small files optimizations

[lemonnierk]

On Wed, May 10, 2017 at 09:14:59AM +0200, Gandalf Corvotempesta wrote:
> Yes much clearer but I think this makes some trouble like space available
> shown by gluster. Or not?

Not really, you'll just see "used space" on your volumes that you won't be
able to track down, keep in mind that the used space on each volume will be
the same (gluster uses the used space from the brick's filesystem)


signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] small files optimizations

[Gandalf Corvotempesta]

Currently, which are the best small files optimization that we can enable
on a gluster storage?

I'm planning to move a couple of dovecot servers, with thousands mail files
(from a couple of KB to less than 10-20MB)

These optimizations are compatible with VMs workload, like sharding?
As gluster doesn't support to "share" bricks with multiple volumes, I would
like to create a single volume with VMs and maildirs

Does it make sense ?

Would be interesting to implement a soft of logical volume (with it's own
feature settings) on top of the same bricks?
Like LVM on a disk.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Quota limits gone after upgrading to 3.8

[Sanoj Unnikrishnan]

Hi Mabi,

Note that limits are still configured and working.
re-adding the limits will not help here (unless you are willing to disable
and re-enable quota first).
The reason is if a gfid exists in quota.conf (because a limit was earlier
set on it), it does not  need change when limit changes.
The quota.conf file only keep track of which gfid have limit set. The
original value of the limits are set in xattr on filesystem

Another work around without mauallly touching quota.conf is,
> Create a new dummy directory anywhere in the FS. add a limit in this
directory.

After this you should be able to see the listing.
If you remove this dummy directory or limit on it, you will once again be
exposed to same issue.

Regards,
Sanoj

On Tue, May 9, 2017 at 10:59 PM, mabi  wrote:

> Hi Sanoj,
>
> Thanks for pointing me at this bug, I was not aware about it.
>
> As this is a production GlusterFS cluster I would rather not mess with the
> quota.conf file as you suggested. Instead I will simply re-add all my
> quotas by running the following command again:
>
> gluster volume quota myvolume limit-usage /directory1 100GB
>
> Can you confirm me that this is safe to run again?
>
>
>
> As soon as I have a minute I will complete your survey about quotas.
>
> Best,
> M.
>
>  Original Message 
> Subject: Re: [Gluster-users] Quota limits gone after upgrading to 3.8
> Local Time: May 9, 2017 6:50 AM
> UTC Time: May 9, 2017 4:50 AM
> From: sunni...@redhat.com
> To: mabi 
> Gluster Users 
>
> Hi mabi,
>
> This bug was fixed recently, https://bugzilla.redhat.com/sh
> ow_bug.cgi?id=1414346. It would be available in 3.11 release. I will plan
> to back port same to earlier releases.
>
> Your quota limits are still set and honored, It is only the listing that
> has gone wrong. Using list with command with single path should display the
> limit on that path. The printing of list gets messed up when the last gfid
> in the quota.conf file is not present in the FS (due to an rmdir without a
> remove limit)
>
> You could use the following workaround to get rid of the issue.
>  => Remove exactly the last 17 bytes of " /var/lib/glusterd/vols/ e>/quota.conf"
>   Note: keep a backup of quota.conf for safety
> If this does not solve the issue, please revert back with
> 1) quota.conf file
> 2) output of list command (when executed along with path)
> 3) getfattr -d -m . -e hex  | grep limit
> It would be great to have your feedback for quota on this thread (
> http://lists.gluster.org/pipermail/gluster-users/2017-April/030676.html)
>
> Thanks & Regards,
> Sanoj
>
>
> On Mon, May 8, 2017 at 7:58 PM, mabi  wrote:
>
>> Hello,
>>
>> I upgraded last week my 2 nodes replica GlusterFS cluster from 3.7.20 to
>> 3.8.11 and on one of the volumes I use the quota feature of GlusterFS.
>> Unfortunately, I just noticed by using the usual command "gluster volume
>> quota myvolume list" that all my quotas on that volume are gone. I had
>> around 10 different quotas set on different directories.
>>
>> Does anyone have an idea where the quotas have vanished? are they gone
>> for always and do I need to re-set them all?
>>
>> Regards,
>> M.
>>
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Is there difference when Nfs-Ganesha is unavailable

[Soumya Koduri]

On 05/10/2017 04:18 AM, ML Wong wrote:

While I m troubleshooting the failover of Nfs-Ganesha, the failover is
always successful when I shutdown Nfs-Ganesha service online while the
OS is running. However, it always failed when I did a either shutdown -r
or power-reset.

During the failure, the Nfs client was just hung. Like you could not do
a "df" or "ls" of the mount point. The share will eventually failover to
the remaining expected node usually after 15 - 20 minutes.


The time taken by pacemaker/corosync services to determine if a node is 
down is usually longer compared to the service down case. But yes it 
should n't take more than couple of minutes.


Could you please check (may be by constantly querying) on how long it 
takes for the virtual-IP to failover by using either 'pcs status' or 'ip 
a' commands. If the IP failover happens quickly but if its just the NFS 
clients taking time to respond, then we have added usage of portblock 
feature to speed up client re-connects post failover. The fixes are 
available (from release-3.9). But before upgrading I suggest to check if 
the delay is with IP failover or client reconnects post failover.


Thanks,
Soumya



Running on Centos7, gluster 3.7.1x, Nfs-Ganesha 2.3.0.x. I currently
don't have the resources to upgrade, but if all of experts here think
that's the only route. I guess I will have to make a case ...

Thanks in advance!


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users