[Gluster-users] Geo-replication benchmarking & DR - failover simulation

[Maurya M]

Hi All,
 As i have now a geo-replication established between my 3 sites , wanted to
test the flow throughput , is there any tools , documentation to measure
the data flow b/w the various slave volumes setup.

Also i understand the replication is unidirectional ( master to slave) - so
incase of DR , are there tested methods to achieve the failover and reverse
the replication and once the primary if restored - go back on the initial
setup.

Appreciate your thoughts & suggestions. Will look forward to the comments
here.

Thanks all,
Maurya
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [ovirt-users] Re: Controller recomandation - LSI2008/9265

[Strahil]

Adding Gluster users' mail list.On Apr 5, 2019 06:02, Leo David 
 wrote:
>
> Hi Everyone,
> Any thoughts on this ?
>
>
> On Wed, Apr 3, 2019, 17:02 Leo David  wrote:
>>
>> Hi Everyone,
>> For a hyperconverged setup started with 3 nodes and going up in time up to 
>> 12 nodes, I have to choose between LSI2008 ( jbod ) and LSI9265 (raid).
>> Perc h710 ( raid ) might be an option too, but on a different chassis.
>> There will not be many disk installed on each node, so the replication will 
>> be replica 3 replicated-distribute volumes across the nodes as:
>> node1/disk1  node2/disk1  node3/disk1
>> node1/disk2  node2/disk2  node3/disk2
>> and so on...
>> As i will add nodes to the cluster ,  I intend expand the volumes using the 
>> same rule.
>> What would it be a better way,  to used jbod cards ( no cache ) or raid card 
>> and create raid0 arrays ( one for each disk ) and therefore have a bit of 
>> raid cache ( 512Mb ) ?
>> Is raid caching a benefit to have it underneath ovirt/gluster as long as I 
>> go for "Jbod"  installation anyway ?
>> Thank you very much !
>> -- 
>> Best regards, Leo David___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Upgrade testing to gluster 6

[Atin Mukherjee]

On Thu, 4 Apr 2019 at 22:10, Darrell Budic  wrote:

> Just the glusterd.log from each node, right?
>

Yes.


>
> On Apr 4, 2019, at 11:25 AM, Atin Mukherjee  wrote:
>
> Darell,
>
> I fully understand that you can't reproduce it and you don't have
> bandwidth to test it again, but would you be able to send us the glusterd
> log from all the nodes when this happened. We would like to go through the
> logs and get back. I would particularly like to see if something has gone
> wrong with transport.socket.listen-port option. But with out the log files
> we can't find out anything. Hope you understand it.
>
> On Thu, Apr 4, 2019 at 9:27 PM Darrell Budic 
> wrote:
>
>> I didn’t follow any specific documents, just a generic rolling upgrade
>> one node at a time. Once the first node didn’t reconnect, I tried to follow
>> the workaround in the bug during the upgrade. Basic procedure was:
>>
>> - take 3 nodes that were initially installed with 3.12.x (forget which,
>> but low number) and had been upgraded directly to 5.5 from 3.12.15
>>   - op-version was 50400
>> - on node A:
>>   - yum install centos-release-gluster6
>>   - yum upgrade (was some ovirt cockpit components, gluster, and a lib or
>> two this time), hit yes
>>   - discover glusterd was dead
>>   - systemctl restart glusterd
>>   - no peer connections, try iptables -F; systemctl restart glusterd, no
>> change
>> - following the workaround in the bug, try iptables -F & restart glusterd
>> on other 2 nodes, no effect
>>   - nodes B & C were still connected to each other and all bricks were
>> fine at this point
>> - try upgrading other 2 nodes and restarting gluster, no effect (iptables
>> still empty)
>>   - lost quota here, so all bricks went offline
>> - read logs, not finding much, but looked at glusterd.vol and compared to
>> new versions
>> - updated glusterd.vol on A and restarted glusterd
>>   - A doesn’t show any connected peers, but both other nodes show A as
>> connected
>> - update glusterd.vol on B & C, restart glusterd
>>   - all nodes show connected and volumes are active and healing
>>
>> The only odd thing in my process was that node A did not have any active
>> bricks on it at the time of the upgrade. It doesn’t seem like this mattered
>> since B & C showed the same symptoms between themselves while being
>> upgraded, but I don’t know. The only log entry that referenced anything
>> about peer connections is included below already.
>>
>> Looks like it was related to my glusterd settings, since that’s what
>> fixed it for me. Unfortunately, I don’t have the bandwidth or the systems
>> to test different versions of that specifically, but maybe you guys can on
>> some test resources? Otherwise, I’ve got another cluster (my production
>> one!) that’s midway through the upgrade from 3.12.15 -> 5.5. I paused when
>> I started getting multiple brick processes on the two nodes that had gone
>> to 5.5 already. I think I’m going to jump the last node right to 6 to try
>> and avoid that mess, and it has the same glusterd.vol settings. I’ll try
>> and capture it’s logs during the upgrade and see if there’s any new info,
>> or if it has the same issues as this group did.
>>
>>   -Darrell
>>
>> On Apr 4, 2019, at 2:54 AM, Sanju Rakonde  wrote:
>>
>> We don't hit https://bugzilla.redhat.com/show_bug.cgi?id=1694010 while
>> upgrading to glusterfs-6. We tested it in different setups and understood
>> that this issue is seen because of some issue in setup.
>>
>> regarding the issue you have faced, can you please let us know which
>> documentation you have followed for the upgrade? During our testing, we
>> didn't hit any such issue. we would like to understand what went wrong.
>>
>> On Thu, Apr 4, 2019 at 2:08 AM Darrell Budic 
>> wrote:
>>
>>> Hari-
>>>
>>> I was upgrading my test cluster from 5.5 to 6 and I hit this bug (
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1694010) or something
>>> similar. In my case, the workaround did not work, and I was left with a
>>> gluster that had gone into no-quorum mode and stopped all the bricks.
>>> Wasn’t much in the logs either, but I noticed my
>>> /etc/glusterfs/glusterd.vol files were not the same as the newer versions,
>>> so I updated them, restarted glusterd, and suddenly the updated node showed
>>> as peer-in-cluster again. Once I updated other notes the same way, things
>>> started working again. Maybe a place to look?
>>>
>>> My old config (all nodes):
>>> volume management
>>> type mgmt/glusterd
>>> option working-directory /var/lib/glusterd
>>> option transport-type socket
>>> option transport.socket.keepalive-time 10
>>> option transport.socket.keepalive-interval 2
>>> option transport.socket.read-fail-log off
>>> option ping-timeout 10
>>> option event-threads 1
>>> option rpc-auth-allow-insecure on
>>> #   option transport.address-family inet6
>>> #   option base-port 49152
>>> end-volume
>>>
>>> changed to:
>>> volume management
>>> type mgmt/glusterd
>>>  

[Gluster-users] Gluster with KVM - VM migration

[Jorge Crespo]

Hi everyone,

First message in this list, hope I can help out as much as I can.

I was wondering if someone could point out any solution already working 
or this would be a matter of scripting.


We are using Gluster for a kind of strange infrastructure , where we 
have let's say 2 NODES , 2 bricks each , 2 volumes total. And both 
servers are mounting as clients both volumes.


We exclusively use these volumes to run VM's. And the reason of the 
infrastructure is to be able to LiveMigrate VM's from one node to the other.


VM's defined and running in NODE 1, MV files in /gluster_gv1 , this 
GlusterFS is also mounted in NODE2 ,but NODE2 doesn't make any real use 
of it.


VM's defined and running in NODE 2, MV files in /gluster_gv2 , as 
before, this GlusterFS is also mounted in NODE1, but it doesn't make any 
real use of it.


So the question comes now:

- Let's say we come to an scenario where NODE 1 comes down. I have the 
VM's files copied to NODE2, I define them in NODE2 and start them , no 
problem with that.


- Now the NODE 1 comes back UP , I guess the safest solution should be 
to have the VM's without Autostart so things don't go messy. But let's 
imagine I want my system to know which VM's are started in NODE 2, and 
start the ones that haven't been started in NODE 2.


Is there any "official" way to achieve this? Basically achieve something 
like vCenter where the cluster keeps track of where the VM's are running 
at any given time, and also being able to start them in a different node 
if their node goes down.


If there is no "official" answer, I'd like to hear your opinions.

Cheers!




smime.p7s
Description: Firma criptográfica S/MIME
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] backupvolfile-server (servers) not working for new mounts?

[Joel Patterson]

I have a gluster 4.1 system with three servers running 
Docker/Kubernetes.    The pods mount filesystems using gluster.


10.13.112.31 is the primary server [A] and all mounts specify it with 
two other servers [10.13.113.116 [B] and 10.13.114.16 [C]] specified in 
backup-volfile-servers.


I'm testing what happens when a server goes down.

If I bring down [B] or [C], no problem, everything restages and works.

But if I bring down [A], any *existing* mount continues to work, but any 
new mounts fail.  I'm seeing messages about all subvolumes being down in 
the pod.


But I've mounted this exact same volume on the same system (before I 
bring down the server) and I can access all the data fine.


Why the failure for new mounts?    I'm on AWS and all servers are in 
different availability zones, but I don't see how that would be an issue.


I tried using just backupvolfile-server and that didn't work either.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Upgrade testing to gluster 6

[Darrell Budic]

Just the glusterd.log from each node, right?

> On Apr 4, 2019, at 11:25 AM, Atin Mukherjee  wrote:
> 
> Darell,
> 
> I fully understand that you can't reproduce it and you don't have bandwidth 
> to test it again, but would you be able to send us the glusterd log from all 
> the nodes when this happened. We would like to go through the logs and get 
> back. I would particularly like to see if something has gone wrong with 
> transport.socket.listen-port option. But with out the log files we can't find 
> out anything. Hope you understand it.
> 
> On Thu, Apr 4, 2019 at 9:27 PM Darrell Budic  > wrote:
> I didn’t follow any specific documents, just a generic rolling upgrade one 
> node at a time. Once the first node didn’t reconnect, I tried to follow the 
> workaround in the bug during the upgrade. Basic procedure was:
> 
> - take 3 nodes that were initially installed with 3.12.x (forget which, but 
> low number) and had been upgraded directly to 5.5 from 3.12.15
>   - op-version was 50400
> - on node A:
>   - yum install centos-release-gluster6
>   - yum upgrade (was some ovirt cockpit components, gluster, and a lib or two 
> this time), hit yes
>   - discover glusterd was dead
>   - systemctl restart glusterd
>   - no peer connections, try iptables -F; systemctl restart glusterd, no 
> change
> - following the workaround in the bug, try iptables -F & restart glusterd on 
> other 2 nodes, no effect
>   - nodes B & C were still connected to each other and all bricks were fine 
> at this point
> - try upgrading other 2 nodes and restarting gluster, no effect (iptables 
> still empty)
>   - lost quota here, so all bricks went offline
> - read logs, not finding much, but looked at glusterd.vol and compared to new 
> versions
> - updated glusterd.vol on A and restarted glusterd
>   - A doesn’t show any connected peers, but both other nodes show A as 
> connected
> - update glusterd.vol on B & C, restart glusterd
>   - all nodes show connected and volumes are active and healing
> 
> The only odd thing in my process was that node A did not have any active 
> bricks on it at the time of the upgrade. It doesn’t seem like this mattered 
> since B & C showed the same symptoms between themselves while being upgraded, 
> but I don’t know. The only log entry that referenced anything about peer 
> connections is included below already.
> 
> Looks like it was related to my glusterd settings, since that’s what fixed it 
> for me. Unfortunately, I don’t have the bandwidth or the systems to test 
> different versions of that specifically, but maybe you guys can on some test 
> resources? Otherwise, I’ve got another cluster (my production one!) that’s 
> midway through the upgrade from 3.12.15 -> 5.5. I paused when I started 
> getting multiple brick processes on the two nodes that had gone to 5.5 
> already. I think I’m going to jump the last node right to 6 to try and avoid 
> that mess, and it has the same glusterd.vol settings. I’ll try and capture 
> it’s logs during the upgrade and see if there’s any new info, or if it has 
> the same issues as this group did.
> 
>   -Darrell
> 
>> On Apr 4, 2019, at 2:54 AM, Sanju Rakonde > > wrote:
>> 
>> We don't hit https://bugzilla.redhat.com/show_bug.cgi?id=1694010 
>>  while upgrading to 
>> glusterfs-6. We tested it in different setups and understood that this issue 
>> is seen because of some issue in setup.
>> 
>> regarding the issue you have faced, can you please let us know which 
>> documentation you have followed for the upgrade? During our testing, we 
>> didn't hit any such issue. we would like to understand what went wrong.
>> 
>> On Thu, Apr 4, 2019 at 2:08 AM Darrell Budic > > wrote:
>> Hari-
>> 
>> I was upgrading my test cluster from 5.5 to 6 and I hit this bug 
>> (https://bugzilla.redhat.com/show_bug.cgi?id=1694010 
>> ) or something similar. 
>> In my case, the workaround did not work, and I was left with a gluster that 
>> had gone into no-quorum mode and stopped all the bricks. Wasn’t much in the 
>> logs either, but I noticed my /etc/glusterfs/glusterd.vol files were not the 
>> same as the newer versions, so I updated them, restarted glusterd, and 
>> suddenly the updated node showed as peer-in-cluster again. Once I updated 
>> other notes the same way, things started working again. Maybe a place to 
>> look?
>> 
>> My old config (all nodes):
>> volume management
>> type mgmt/glusterd
>> option working-directory /var/lib/glusterd
>> option transport-type socket
>> option transport.socket.keepalive-time 10
>> option transport.socket.keepalive-interval 2
>> option transport.socket.read-fail-log off
>> option ping-timeout 10
>> option event-threads 1
>> option rpc-auth-allow-insecure on
>> #   option transport.address-family inet6
>>

Re: [Gluster-users] [Gluster-devel] Upgrade testing to gluster 6

[Atin Mukherjee]

Darell,

I fully understand that you can't reproduce it and you don't have bandwidth
to test it again, but would you be able to send us the glusterd log from
all the nodes when this happened. We would like to go through the logs and
get back. I would particularly like to see if something has gone wrong with
transport.socket.listen-port option. But with out the log files we can't
find out anything. Hope you understand it.

On Thu, Apr 4, 2019 at 9:27 PM Darrell Budic  wrote:

> I didn’t follow any specific documents, just a generic rolling upgrade one
> node at a time. Once the first node didn’t reconnect, I tried to follow the
> workaround in the bug during the upgrade. Basic procedure was:
>
> - take 3 nodes that were initially installed with 3.12.x (forget which,
> but low number) and had been upgraded directly to 5.5 from 3.12.15
>   - op-version was 50400
> - on node A:
>   - yum install centos-release-gluster6
>   - yum upgrade (was some ovirt cockpit components, gluster, and a lib or
> two this time), hit yes
>   - discover glusterd was dead
>   - systemctl restart glusterd
>   - no peer connections, try iptables -F; systemctl restart glusterd, no
> change
> - following the workaround in the bug, try iptables -F & restart glusterd
> on other 2 nodes, no effect
>   - nodes B & C were still connected to each other and all bricks were
> fine at this point
> - try upgrading other 2 nodes and restarting gluster, no effect (iptables
> still empty)
>   - lost quota here, so all bricks went offline
> - read logs, not finding much, but looked at glusterd.vol and compared to
> new versions
> - updated glusterd.vol on A and restarted glusterd
>   - A doesn’t show any connected peers, but both other nodes show A as
> connected
> - update glusterd.vol on B & C, restart glusterd
>   - all nodes show connected and volumes are active and healing
>
> The only odd thing in my process was that node A did not have any active
> bricks on it at the time of the upgrade. It doesn’t seem like this mattered
> since B & C showed the same symptoms between themselves while being
> upgraded, but I don’t know. The only log entry that referenced anything
> about peer connections is included below already.
>
> Looks like it was related to my glusterd settings, since that’s what fixed
> it for me. Unfortunately, I don’t have the bandwidth or the systems to test
> different versions of that specifically, but maybe you guys can on some
> test resources? Otherwise, I’ve got another cluster (my production one!)
> that’s midway through the upgrade from 3.12.15 -> 5.5. I paused when I
> started getting multiple brick processes on the two nodes that had gone to
> 5.5 already. I think I’m going to jump the last node right to 6 to try and
> avoid that mess, and it has the same glusterd.vol settings. I’ll try and
> capture it’s logs during the upgrade and see if there’s any new info, or if
> it has the same issues as this group did.
>
>   -Darrell
>
> On Apr 4, 2019, at 2:54 AM, Sanju Rakonde  wrote:
>
> We don't hit https://bugzilla.redhat.com/show_bug.cgi?id=1694010 while
> upgrading to glusterfs-6. We tested it in different setups and understood
> that this issue is seen because of some issue in setup.
>
> regarding the issue you have faced, can you please let us know which
> documentation you have followed for the upgrade? During our testing, we
> didn't hit any such issue. we would like to understand what went wrong.
>
> On Thu, Apr 4, 2019 at 2:08 AM Darrell Budic 
> wrote:
>
>> Hari-
>>
>> I was upgrading my test cluster from 5.5 to 6 and I hit this bug (
>> https://bugzilla.redhat.com/show_bug.cgi?id=1694010) or something
>> similar. In my case, the workaround did not work, and I was left with a
>> gluster that had gone into no-quorum mode and stopped all the bricks.
>> Wasn’t much in the logs either, but I noticed my
>> /etc/glusterfs/glusterd.vol files were not the same as the newer versions,
>> so I updated them, restarted glusterd, and suddenly the updated node showed
>> as peer-in-cluster again. Once I updated other notes the same way, things
>> started working again. Maybe a place to look?
>>
>> My old config (all nodes):
>> volume management
>> type mgmt/glusterd
>> option working-directory /var/lib/glusterd
>> option transport-type socket
>> option transport.socket.keepalive-time 10
>> option transport.socket.keepalive-interval 2
>> option transport.socket.read-fail-log off
>> option ping-timeout 10
>> option event-threads 1
>> option rpc-auth-allow-insecure on
>> #   option transport.address-family inet6
>> #   option base-port 49152
>> end-volume
>>
>> changed to:
>> volume management
>> type mgmt/glusterd
>> option working-directory /var/lib/glusterd
>> option transport-type socket,rdma
>> option transport.socket.keepalive-time 10
>> option transport.socket.keepalive-interval 2
>> option transport.socket.read-fail-log off
>> option transport.socket.listen-port 2

Re: [Gluster-users] [Gluster-devel] Upgrade testing to gluster 6

[Darrell Budic]

I didn’t follow any specific documents, just a generic rolling upgrade one node 
at a time. Once the first node didn’t reconnect, I tried to follow the 
workaround in the bug during the upgrade. Basic procedure was:

- take 3 nodes that were initially installed with 3.12.x (forget which, but low 
number) and had been upgraded directly to 5.5 from 3.12.15
  - op-version was 50400
- on node A:
  - yum install centos-release-gluster6
  - yum upgrade (was some ovirt cockpit components, gluster, and a lib or two 
this time), hit yes
  - discover glusterd was dead
  - systemctl restart glusterd
  - no peer connections, try iptables -F; systemctl restart glusterd, no change
- following the workaround in the bug, try iptables -F & restart glusterd on 
other 2 nodes, no effect
  - nodes B & C were still connected to each other and all bricks were fine at 
this point
- try upgrading other 2 nodes and restarting gluster, no effect (iptables still 
empty)
  - lost quota here, so all bricks went offline
- read logs, not finding much, but looked at glusterd.vol and compared to new 
versions
- updated glusterd.vol on A and restarted glusterd
  - A doesn’t show any connected peers, but both other nodes show A as connected
- update glusterd.vol on B & C, restart glusterd
  - all nodes show connected and volumes are active and healing

The only odd thing in my process was that node A did not have any active bricks 
on it at the time of the upgrade. It doesn’t seem like this mattered since B & 
C showed the same symptoms between themselves while being upgraded, but I don’t 
know. The only log entry that referenced anything about peer connections is 
included below already.

Looks like it was related to my glusterd settings, since that’s what fixed it 
for me. Unfortunately, I don’t have the bandwidth or the systems to test 
different versions of that specifically, but maybe you guys can on some test 
resources? Otherwise, I’ve got another cluster (my production one!) that’s 
midway through the upgrade from 3.12.15 -> 5.5. I paused when I started getting 
multiple brick processes on the two nodes that had gone to 5.5 already. I think 
I’m going to jump the last node right to 6 to try and avoid that mess, and it 
has the same glusterd.vol settings. I’ll try and capture it’s logs during the 
upgrade and see if there’s any new info, or if it has the same issues as this 
group did.

  -Darrell

> On Apr 4, 2019, at 2:54 AM, Sanju Rakonde  wrote:
> 
> We don't hit https://bugzilla.redhat.com/show_bug.cgi?id=1694010 
>  while upgrading to 
> glusterfs-6. We tested it in different setups and understood that this issue 
> is seen because of some issue in setup.
> 
> regarding the issue you have faced, can you please let us know which 
> documentation you have followed for the upgrade? During our testing, we 
> didn't hit any such issue. we would like to understand what went wrong.
> 
> On Thu, Apr 4, 2019 at 2:08 AM Darrell Budic  > wrote:
> Hari-
> 
> I was upgrading my test cluster from 5.5 to 6 and I hit this bug 
> (https://bugzilla.redhat.com/show_bug.cgi?id=1694010 
> ) or something similar. 
> In my case, the workaround did not work, and I was left with a gluster that 
> had gone into no-quorum mode and stopped all the bricks. Wasn’t much in the 
> logs either, but I noticed my /etc/glusterfs/glusterd.vol files were not the 
> same as the newer versions, so I updated them, restarted glusterd, and 
> suddenly the updated node showed as peer-in-cluster again. Once I updated 
> other notes the same way, things started working again. Maybe a place to look?
> 
> My old config (all nodes):
> volume management
> type mgmt/glusterd
> option working-directory /var/lib/glusterd
> option transport-type socket
> option transport.socket.keepalive-time 10
> option transport.socket.keepalive-interval 2
> option transport.socket.read-fail-log off
> option ping-timeout 10
> option event-threads 1
> option rpc-auth-allow-insecure on
> #   option transport.address-family inet6
> #   option base-port 49152
> end-volume
> 
> changed to:
> volume management
> type mgmt/glusterd
> option working-directory /var/lib/glusterd
> option transport-type socket,rdma
> option transport.socket.keepalive-time 10
> option transport.socket.keepalive-interval 2
> option transport.socket.read-fail-log off
> option transport.socket.listen-port 24007
> option transport.rdma.listen-port 24008
> option ping-timeout 0
> option event-threads 1
> option rpc-auth-allow-insecure on
> #   option lock-timer 180
> #   option transport.address-family inet6
> #   option base-port 49152
> option max-port  60999
> end-volume
> 
> the only thing I found in the glusterd logs that looks relevant was (repeated 
> for both of the other nodes in this cluster), so no cl

Re: [Gluster-users] thin arbiter setup

[Ashish Pandey]

Hi, 

Currently, thin-arbiter can be setup using GD2. glustercli command is provided 
by GD2 only. 
Have you installed and started GD2 first? 
Could you please mention in which step you faced issue? 

--- 
Ashish 

- Original Message -

From: "banda bassotti"  
To: gluster-users@gluster.org 
Sent: Thursday, April 4, 2019 3:45:01 PM 
Subject: [Gluster-users] thin arbiter setup 

Hi all, is there a detailed guide on how to configure a two-node cluster with a 
thin arbiter? I tried to follow the guide: 

https://docs.gluster.org/en/latest/Administrator%20Guide/Thin-Arbiter-Volumes/#setting-up-thin-arbiter-volume
 

but it doesn't work. I'm using debian stretch and gluster 6 repository. 

thnx a lot. 

banda. 

___ 
Gluster-users mailing list 
Gluster-users@gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users 

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] thin arbiter setup

[banda bassotti]

Hi all, is there a detailed guide on how to configure a two-node cluster
with a thin arbiter? I tried to follow the guide:

https://docs.gluster.org/en/latest/Administrator%20Guide/Thin-Arbiter-Volumes/#setting-up-thin-arbiter-volume


but it doesn't work.  I'm using debian stretch and gluster 6 repository.

thnx a lot.

banda.
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] performance - what can I expect

[Pascal Suter]

I just noticed i left the most important parameters out :)

here's the write command with filesize and recordsize in it as well :)

./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w 
-+S 0 -s 200G -r 16384k


also i ran the benchmark without direct_io which resulted in an even 
worse performance.


i also tried to mount the gluster volume via nfs-ganesha which further 
reduced throughput down to about 450MB/s


if i run the iozone benchmark with 3 threads writing to all three bricks 
directly (from the xfs filesystem) i get throughputs of around 6GB/s .. 
if I run the same benchmark through gluster mounted locally using the 
fuse client and with enough threads so that each brick gets at least one 
file written to it, i end up seing throughputs around 1.5GB/s .. that's 
a 4x decrease in performance. at it actually is the same if i run the 
benchmark with less threads and files only get written to two out of 
three bricks.


cpu load on the server is around 25% by the way, nicely distributed 
across all available cores.


i can't believe that gluster should really be so slow and everybody is 
just happily using it. any hints on what i'm doing wrong are very welcome.


i'm using gluster 6.0 by the way.

regards

Pascal

On 03.04.19 12:28, Pascal Suter wrote:

Hi all

I am currently testing gluster on a single server. I have three 
bricks, each a hardware RAID6 volume with thin provisioned LVM that 
was aligned to the RAID and then formatted with xfs.


i've created a distributed volume so that entire files get distributed 
across my three bricks.


first I ran a iozone benchmark across each brick testing the read and 
write perofrmance of a single large file per brick


i then mounted my gluster volume locally and ran another iozone run 
with the same parameters writing a single file. the file went to brick 
1 which, when used driectly, would write with 2.3GB/s and read with 
1.5GB/s. however, through gluster i got only 800MB/s read and 750MB/s 
write throughput


another run with two processes each writing a file, where one file 
went to the first brick and the other file to the second brick (which 
by itself when directly accessed wrote at 2.8GB/s and read at 2.7GB/s) 
resulted in 1.2GB/s of aggregated write and also aggregated read 
throughput.


Is this a normal performance i can expect out of a glusterfs or is it 
worth tuning in order to really get closer to the actual brick 
filesystem performance?


here are the iozone commands i use for writing and reading.. note that 
i am using directIO in order to make sure i don't get fooled by cache :)


./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 
-s $filesize -r $recordsize > iozone-brick${b}-write.txt


./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 
-s $filesize -r $recordsize > iozone-brick${b}-read.txt


cheers

Pascal

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Enabling quotas on gluster

[Hari Gowtham]

Hi,

The performance hit that quota causes depended on a number of factors
like:
1) the number of files,
2) the depth of the directories in the FS
3) the breadth of the directories in the FS
4) the number of bricks.

These are the main contributions to the performance hit.
If the volume is of lesser size then quota should work fine.
Let us know more about your use case to help you better.

Note: gluster quota is not being actively worked on.

On Thu, Apr 4, 2019 at 3:45 AM Lindolfo Meira  wrote:
>
> Hi folks.
>
> Does anyone know how significant is the performance penalty for enabling
> directory level quotas on a gluster fs, compared to the case with no
> quotas at all?
>
>
> Lindolfo Meira, MSc
> Diretor Geral, Centro Nacional de Supercomputação
> Universidade Federal do Rio Grande do Sul
> +55 (51) 3308-3139___
> Gluster-users mailing list
> Gluster-users@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users



-- 
Regards,
Hari Gowtham.
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] [Gluster-devel] Upgrade testing to gluster 6

[Sanju Rakonde]

We don't hit https://bugzilla.redhat.com/show_bug.cgi?id=1694010 while
upgrading to glusterfs-6. We tested it in different setups and understood
that this issue is seen because of some issue in setup.

regarding the issue you have faced, can you please let us know which
documentation you have followed for the upgrade? During our testing, we
didn't hit any such issue. we would like to understand what went wrong.

On Thu, Apr 4, 2019 at 2:08 AM Darrell Budic  wrote:

> Hari-
>
> I was upgrading my test cluster from 5.5 to 6 and I hit this bug (
> https://bugzilla.redhat.com/show_bug.cgi?id=1694010) or something
> similar. In my case, the workaround did not work, and I was left with a
> gluster that had gone into no-quorum mode and stopped all the bricks.
> Wasn’t much in the logs either, but I noticed my
> /etc/glusterfs/glusterd.vol files were not the same as the newer versions,
> so I updated them, restarted glusterd, and suddenly the updated node showed
> as peer-in-cluster again. Once I updated other notes the same way, things
> started working again. Maybe a place to look?
>
> My old config (all nodes):
> volume management
> type mgmt/glusterd
> option working-directory /var/lib/glusterd
> option transport-type socket
> option transport.socket.keepalive-time 10
> option transport.socket.keepalive-interval 2
> option transport.socket.read-fail-log off
> option ping-timeout 10
> option event-threads 1
> option rpc-auth-allow-insecure on
> #   option transport.address-family inet6
> #   option base-port 49152
> end-volume
>
> changed to:
> volume management
> type mgmt/glusterd
> option working-directory /var/lib/glusterd
> option transport-type socket,rdma
> option transport.socket.keepalive-time 10
> option transport.socket.keepalive-interval 2
> option transport.socket.read-fail-log off
> option transport.socket.listen-port 24007
> option transport.rdma.listen-port 24008
> option ping-timeout 0
> option event-threads 1
> option rpc-auth-allow-insecure on
> #   option lock-timer 180
> #   option transport.address-family inet6
> #   option base-port 49152
> option max-port  60999
> end-volume
>
> the only thing I found in the glusterd logs that looks relevant was
> (repeated for both of the other nodes in this cluster), so no clue why it
> happened:
> [2019-04-03 20:19:16.802638] I [MSGID: 106004]
> [glusterd-handler.c:6427:__glusterd_peer_rpc_notify] 0-management: Peer
>  (<0ecbf953-681b-448f-9746-d1c1fe7a0978>), in state  Cluster>, has disconnected from glusterd.
>
>
> On Apr 2, 2019, at 4:53 AM, Atin Mukherjee 
> wrote:
>
>
>
> On Mon, 1 Apr 2019 at 10:28, Hari Gowtham  wrote:
>
>> Comments inline.
>>
>> On Mon, Apr 1, 2019 at 5:55 AM Sankarshan Mukhopadhyay
>>  wrote:
>> >
>> > Quite a considerable amount of detail here. Thank you!
>> >
>> > On Fri, Mar 29, 2019 at 11:42 AM Hari Gowtham 
>> wrote:
>> > >
>> > > Hello Gluster users,
>> > >
>> > > As you all aware that glusterfs-6 is out, we would like to inform you
>> > > that, we have spent a significant amount of time in testing
>> > > glusterfs-6 in upgrade scenarios. We have done upgrade testing to
>> > > glusterfs-6 from various releases like 3.12, 4.1 and 5.3.
>> > >
>> > > As glusterfs-6 has got in a lot of changes, we wanted to test those
>> portions.
>> > > There were xlators (and respective options to enable/disable them)
>> > > added and deprecated in glusterfs-6 from various versions [1].
>> > >
>> > > We had to check the following upgrade scenarios for all such options
>> > > Identified in [1]:
>> > > 1) option never enabled and upgraded
>> > > 2) option enabled and then upgraded
>> > > 3) option enabled and then disabled and then upgraded
>> > >
>> > > We weren't manually able to check all the combinations for all the
>> options.
>> > > So the options involving enabling and disabling xlators were
>> prioritized.
>> > > The below are the result of the ones tested.
>> > >
>> > > Never enabled and upgraded:
>> > > checked from 3.12, 4.1, 5.3 to 6 the upgrade works.
>> > >
>> > > Enabled and upgraded:
>> > > Tested for tier which is deprecated, It is not a recommended upgrade.
>> > > As expected the volume won't be consumable and will have a few more
>> > > issues as well.
>> > > Tested with 3.12, 4.1 and 5.3 to 6 upgrade.
>> > >
>> > > Enabled, disabled before upgrade.
>> > > Tested for tier with 3.12 and the upgrade went fine.
>> > >
>> > > There is one common issue to note in every upgrade. The node being
>> > > upgraded is going into disconnected state. You have to flush the
>> iptables
>> > > and the restart glusterd on all nodes to fix this.
>> > >
>> >
>> > Is this something that is written in the upgrade notes? I do not seem
>> > to recall, if not, I'll send a PR
>>
>> No this wasn't mentioned in the release notes. PRs are welcome.
>>
>> >
>> > > The testing for enabling new options is still pending. The new options
>> > > won't cause as much issues as

Re: [Gluster-users] Gluster 5.5 slower than 3.12.15

[Strahil]

Hi Amar,

I would like to test Cluster v6 , but as I'm quite new to oVirt - I'm not sure 
if oVirt <-> Gluster will communicate  properly

Did anyone test rollback from v6 to v5.5 ? If rollback is possible - I would be 
happy to give it a try.

Best Regards,
Strahil NikolovOn Apr 3, 2019 11:35, Amar Tumballi Suryanarayan 
 wrote:
>
> Strahil,
>
> With some basic testing, we are noticing the similar behavior too.
>
> One of the issue we identified was increased n/w usage in 5.x series (being 
> addressed by https://review.gluster.org/#/c/glusterfs/+/22404/), and there 
> are few other features which write extended attributes which caused some 
> delay.
>
> We are in the process of publishing some numbers with release-3.12.x, 
> release-5 and release-6 comparison soon. With some numbers we are already 
> seeing release-6 currently is giving really good performance in many 
> configurations, specially for 1x3 replicate volume type.
>
> While we continue to identify and fix issues in 5.x series, one of the 
> request is to validate release-6.x (6.0 or 6.1 which would happen on April 
> 10th), so you can see the difference in your workload.
>
> Regards,
> Amar
>
>
>
> On Wed, Apr 3, 2019 at 5:57 AM Strahil Nikolov  wrote:
>>
>> Hi Community,
>>
>> I have the feeling that with gluster v5.5 I have poorer performance than it 
>> used to be on 3.12.15. Did you observe something like that?
>>
>> I have a 3 node Hyperconverged Cluster (ovirt + glusterfs with replica 3 
>> arbiter1 volumes) with NFS Ganesha and since I have upgraded to v5 - the 
>> issues came up.
>> First it was 5.3 notorious experience and now with 5.5 - my sanlock is 
>> having problems and higher latency than it used to be. I have switched from 
>> NFS-Ganesha to pure FUSE , but the latency problems do not go away.
>>
>> Of course , this is partially due to the consumer hardware, but as the 
>> hardware has not changed I was hoping that the performance will remain as is.
>>
>> So, do you expect 5.5 to perform less than 3.12 ?
>>
>> Some info:
>> Volume Name: engine
>> Type: Replicate
>> Volume ID: 30ca1cc2-f2f7-4749-9e2e-cee9d7099ded
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: ovirt1:/gluster_bricks/engine/engine
>> Brick2: ovirt2:/gluster_bricks/engine/engine
>> Brick3: ovirt3:/gluster_bricks/engine/engine (arbiter)
>> Options Reconfigured:
>> performance.client-io-threads: off
>> nfs.disable: on
>> transport.address-family: inet
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.low-prio-threads: 32
>> network.remote-dio: off
>> cluster.eager-lock: enable
>> cluster.quorum-type: auto
>> cluster.server-quorum-type: server
>> cluster.data-self-heal-algorithm: full
>> cluster.locking-scheme: granular
>> cluster.shd-max-threads: 8
>> cluster.shd-wait-qlength: 1
>> features.shard: on
>> user.cifs: off
>> storage.owner-uid: 36
>> storage.owner-gid: 36
>> network.ping-timeout: 30
>> performance.strict-o-direct: on
>> cluster.granular-entry-heal: enable
>> cluster.enable-shared-storage: enable
>>
>> Network: 1 gbit/s
>>
>> Filesystem:XFS
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> -- 
> Amar Tumballi (amarts)
___
Gluster-users mailing list
Gluster-users@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users