Re: [Gluster-users] GlusterFS and Kafka

[Christopher Schmidt]

Well, a "turning off caching" warning is ok.
But without doing anything, Kafka definitely doesn't work. Which is somehow
strange because it is a normal (IMHO) JVM process written in Scala. I am
wondering if there are some issues with other tools too.

Vijay Bellur  schrieb am Di., 23. Mai 2017 um 05:48 Uhr:

> On Mon, May 22, 2017 at 11:49 AM, Joe Julian  wrote:
>
>> This may be asking too much, but can you explain why or how it's even
>> possible to bypass the cache like this, Vijay?
>>
>
> This is a good question and the answers to that is something I should have
> elaborated a bit more in my previous response. As far as the why is
> concerned, gluster's client stack is configured by default to provide
> reasonable performance and not  be very strongly consistent for
> applications that need the most accurate metadata for their functioning.
> Turning off the performance translators provide more stronger consistency
> and we have seen applications that rely on a high degree of consistency
> work better with that configuration. It is with this backdrop I suggested
> performance translators be turned off from the client stack for Kafka.
>
> On how it is possible, the translator model of gluster helps us to enable
> or disable optional functionality from the stack. There is no single
> configuration that can accommodate workloads with varying profiles and
> having a modular architecture is a plus for gluster - the storage stack can
> be tuned to suit varying application profiles. We are exploring the
> possibility of providing custom profiles (collections of options) for
> popular applications to make it easier for all of us. Note that disabling
> performance translators in gluster is similar to turning off caching with
> the NFS client. In parallel we are also looking to alter the behavior of
> performance translators to provide as much consistency as possible by
> default.
>
> Thanks,
> Vijay
>
>>
>>
>> On May 22, 2017 7:41:40 AM PDT, Vijay Bellur  wrote:
>>>
>>> Looks like a problem with caching. Can you please try by disabling all
>>> performance translators? The following configuration commands would disable
>>> performance translators in the gluster client stack:
>>>
>>> gluster volume set  performance.quick-read off
>>> gluster volume set  performance.io-cache off
>>> gluster volume set  performance.write-behind off
>>> gluster volume set  performance.stat-prefetch off
>>> gluster volume set  performance.read-ahead off
>>> gluster volume set  performance.readdir-ahead off
>>> gluster volume set  performance.open-behind off
>>> gluster volume set  performance.client-io-threads off
>>>
>>> Thanks,
>>> Vijay
>>>
>>>
>>>
>>> On Mon, May 22, 2017 at 9:46 AM, Christopher Schmidt >> > wrote:
>>>
 Hi all,

 has anyone ever successfully deployed a Kafka (Cluster) on GlusterFS
 volumes?

 I my case it's a Kafka Kubernetes-StatefulSet and a Heketi GlusterFS.
 Needless to say that I am getting a lot of filesystem related
 exceptions like this one:

 Failed to read `log header` from file channel
 `sun.nio.ch.FileChannelImpl@67afa54a`. Expected to read 12 bytes, but
 reached end of file after reading 0 bytes. Started read from position
 123065680.

 I limited the amount of exceptions with
 the log.flush.interval.messages=1 option, but not all...

 best Christopher


 ___
 Gluster-users mailing list
 Gluster-users@gluster.org
 http://lists.gluster.org/mailman/listinfo/gluster-users

>>>
>>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] GlusterFS and Kafka

[Christopher Schmidt]

OK, seems that this works now.

A couple of questions:
- What do you think, are all these options necessary for Kafka?
- You wrote that there have to be kind of application profiles. So to find
out, which set of options work is currently a matter of testing (and hope)?
Or are there any experiences for MongoDB / ProstgreSQL / Zookeeper etc.?
- I am using Heketi and Dynamik Storage Provisioning together with
Kubernetes. Can I set this volume options somehow by default or by volume
plugin?

Thanks for you help... really appreciated.. Christopher

Vijay Bellur  schrieb am Mo., 22. Mai 2017 um 16:41 Uhr:

> Looks like a problem with caching. Can you please try by disabling all
> performance translators? The following configuration commands would disable
> performance translators in the gluster client stack:
>
> gluster volume set  performance.quick-read off
> gluster volume set  performance.io-cache off
> gluster volume set  performance.write-behind off
> gluster volume set  performance.stat-prefetch off
> gluster volume set  performance.read-ahead off
> gluster volume set  performance.readdir-ahead off
> gluster volume set  performance.open-behind off
> gluster volume set  performance.client-io-threads off
>
> Thanks,
> Vijay
>
>
>
> On Mon, May 22, 2017 at 9:46 AM, Christopher Schmidt 
> wrote:
>
>> Hi all,
>>
>> has anyone ever successfully deployed a Kafka (Cluster) on GlusterFS
>> volumes?
>>
>> I my case it's a Kafka Kubernetes-StatefulSet and a Heketi GlusterFS.
>> Needless to say that I am getting a lot of filesystem related exceptions
>> like this one:
>>
>> Failed to read `log header` from file channel
>> `sun.nio.ch.FileChannelImpl@67afa54a`. Expected to read 12 bytes, but
>> reached end of file after reading 0 bytes. Started read from position
>> 123065680.
>>
>> I limited the amount of exceptions with the log.flush.interval.messages=1
>> option, but not all...
>>
>> best Christopher
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] GlusterFS and Kafka

[Vijay Bellur]

On Mon, May 22, 2017 at 11:49 AM, Joe Julian  wrote:

> This may be asking too much, but can you explain why or how it's even
> possible to bypass the cache like this, Vijay?
>

This is a good question and the answers to that is something I should have
elaborated a bit more in my previous response. As far as the why is
concerned, gluster's client stack is configured by default to provide
reasonable performance and not  be very strongly consistent for
applications that need the most accurate metadata for their functioning.
Turning off the performance translators provide more stronger consistency
and we have seen applications that rely on a high degree of consistency
work better with that configuration. It is with this backdrop I suggested
performance translators be turned off from the client stack for Kafka.

On how it is possible, the translator model of gluster helps us to enable
or disable optional functionality from the stack. There is no single
configuration that can accommodate workloads with varying profiles and
having a modular architecture is a plus for gluster - the storage stack can
be tuned to suit varying application profiles. We are exploring the
possibility of providing custom profiles (collections of options) for
popular applications to make it easier for all of us. Note that disabling
performance translators in gluster is similar to turning off caching with
the NFS client. In parallel we are also looking to alter the behavior of
performance translators to provide as much consistency as possible by
default.

Thanks,
Vijay

>
>
> On May 22, 2017 7:41:40 AM PDT, Vijay Bellur  wrote:
>>
>> Looks like a problem with caching. Can you please try by disabling all
>> performance translators? The following configuration commands would disable
>> performance translators in the gluster client stack:
>>
>> gluster volume set  performance.quick-read off
>> gluster volume set  performance.io-cache off
>> gluster volume set  performance.write-behind off
>> gluster volume set  performance.stat-prefetch off
>> gluster volume set  performance.read-ahead off
>> gluster volume set  performance.readdir-ahead off
>> gluster volume set  performance.open-behind off
>> gluster volume set  performance.client-io-threads off
>>
>> Thanks,
>> Vijay
>>
>>
>>
>> On Mon, May 22, 2017 at 9:46 AM, Christopher Schmidt 
>> wrote:
>>
>>> Hi all,
>>>
>>> has anyone ever successfully deployed a Kafka (Cluster) on GlusterFS
>>> volumes?
>>>
>>> I my case it's a Kafka Kubernetes-StatefulSet and a Heketi GlusterFS.
>>> Needless to say that I am getting a lot of filesystem related exceptions
>>> like this one:
>>>
>>> Failed to read `log header` from file channel
>>> `sun.nio.ch.FileChannelImpl@67afa54a`. Expected to read 12 bytes, but
>>> reached end of file after reading 0 bytes. Started read from position
>>> 123065680.
>>>
>>> I limited the amount of exceptions with the log.flush.interval.messages=1
>>> option, but not all...
>>>
>>> best Christopher
>>>
>>>
>>> ___
>>> Gluster-users mailing list
>>> Gluster-users@gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Release 3.11: RC1 is tagged!

[Shyam]

Hi,

We just finished tagging release 3.11.0 RC1, that contains a few more 
fixes post 3.11.0 RC0.


Packages for the same will be made available soon.

*Attention* contributors:
  - We still are to see any updates to the release-notes [2], we have a 
week before the release, please update the release notes appropriately


*Attention* all:
  - Any pending bugs that are critical to the release need to be marked 
as a blocker against [1] and we have about a week left to close out 
these bugs!


Thanks,
Shyam

[1] Tracker BZ for 3.11.0 blockers:
https://bugzilla.redhat.com/show_bug.cgi?id=glusterfs-3.11.0

[2] Release notes: 
https://github.com/gluster/glusterfs/blob/release-3.11/doc/release-notes/3.11.0.md

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Quorum lost with single failed peer in rep 3...?

[Gambit15]

Hey guys,
 I use a replica 3 arbiter 1 setup for hosting VMs, and have just had an
issue where taking one of the non-arbiter peers offline caused gluster to
complain of lost quorum & pause the volumes.

The two "full" peers host the VMs and data, and the arbiter is a VM on a
neighbouring cluster.

Before taking the peer offline, I migrated all VMs from it, verified there
were no running heal processes, and that all peers were connected.

Quorum is configured with the following...
cluster.server-quorum-type: server
cluster.quorum-type: auto

As I understand it, quorum auto means 51%, so quorum should be maintained
if any one of the peers fails. There have been a couple of occasions where
the arbiter went offline, but quorum was maintained as expected &
everything continued to function.

When the volumes were paused, I connected to the remaining node to see what
was going on. "gluster peer status" reported the offline node as
disconnected & the arbiter as connected, as expected. All "gluster volume"
commands hung.
When the offline node was rebooted, quorum returned & all services resumed.

>From the logs (pasted below), it appears the primary node & the arbiter
disconnected from each other around the time the secondary node went
offline, although that's contrary to what was reported by "gluster peer
status".

s0, 10.123.123.10: Full peer
s1, 10.123.123.11: Full peer (taken offline)
s2, 10.123.123.12: Arbiter


= s0 =

[2017-05-22 18:24:20.854775] I [MSGID: 106163]
[glusterd-handshake.c:1271:__glusterd_mgmt_hndsk_versions_ack]
0-management: using the op-version 30800
[2017-05-22 18:31:30.398272] E [rpc-clnt.c:200:call_bail] 0-management:
bailing out frame type(Peer mgmt) op(--(2)) xid = 0x7ab6 sent = 2017-05-22
18:21:20.549877. timeout = 600 for 10.123.123.12:2
4007
[2017-05-22 18:35:20.420878] E [rpc-clnt.c:200:call_bail] 0-management:
bailing out frame type(glusterd mgmt) op(--(3)) xid = 0x7ab7 sent =
2017-05-22 18:25:11.187323. timeout = 600 for 10.123.123.
12:24007
[2017-05-22 18:35:20.420943] E [MSGID: 106153]
[glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on s2.
Please check log file for details.
[2017-05-22 18:35:20.421103] I [socket.c:3465:socket_submit_reply]
0-socket.management: not connected (priv->connected = -1)
[2017-05-22 18:35:20.421126] E [rpcsvc.c:1325:rpcsvc_submit_generic]
0-rpc-service: failed to submit message (XID: 0x1, Program: GlusterD svc
cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.ma
nagement)
[2017-05-22 18:35:20.421145] E [MSGID: 106430]
[glusterd-utils.c:470:glusterd_submit_reply] 0-glusterd: Reply submission
failed
[2017-05-22 18:36:24.732098] W [socket.c:590:__socket_rwv] 0-management:
readv on 10.123.123.11:24007 failed (Não há dados disponíveis)
[2017-05-22 18:36:24.732214] I [MSGID: 106004]
[glusterd-handler.c:5219:__glusterd_peer_rpc_notify] 0-management: Peer
 (), in state ,
has
 disconnected from glusterd.
[2017-05-22 18:36:24.732293] W
[glusterd-locks.c:675:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/3.8.5/xlator/mgmt/glusterd.so(+0x1de5c)
[0x7f17ae400e5c] -->/usr/lib64/glusterfs/3.8.5/xlator/
mgmt/glusterd.so(+0x27a08) [0x7f17ae40aa08]
-->/usr/lib64/glusterfs/3.8.5/xlator/mgmt/glusterd.so(+0xd07fa)
[0x7f17ae4b37fa] ) 0-management: Lock for vol data not held
[2017-05-22 18:36:24.732303] W [MSGID: 106118]
[glusterd-handler.c:5241:__glusterd_peer_rpc_notify] 0-management: Lock not
released for data
[2017-05-22 18:36:24.732323] W
[glusterd-locks.c:675:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/3.8.5/xlator/mgmt/glusterd.so(+0x1de5c)
[0x7f17ae400e5c] -->/usr/lib64/glusterfs/3.8.5/xlator/
mgmt/glusterd.so(+0x27a08) [0x7f17ae40aa08]
-->/usr/lib64/glusterfs/3.8.5/xlator/mgmt/glusterd.so(+0xd07fa)
[0x7f17ae4b37fa] ) 0-management: Lock for vol engine not held
[2017-05-22 18:36:24.732330] W [MSGID: 106118]
[glusterd-handler.c:5241:__glusterd_peer_rpc_notify] 0-management: Lock not
released for engine
[2017-05-22 18:36:24.732350] W
[glusterd-locks.c:675:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/3.8.5/xlator/mgmt/glusterd.so(+0x1de5c)
[0x7f17ae400e5c]
-->/usr/lib64/glusterfs/3.8.5/xlator/mgmt/glusterd.so(+0x27a08)
[0x7f17ae40aa08]
-->/usr/lib64/glusterfs/3.8.5/xlator/mgmt/glusterd.so(+0xd07fa)
[0x7f17ae4b37fa] ) 0-management: Lock for vol iso not held
[2017-05-22 18:36:24.732382] W [MSGID: 106118]
[glusterd-handler.c:5241:__glusterd_peer_rpc_notify] 0-management: Lock not
released for iso
[2017-05-22 18:36:24.732405] C [MSGID: 106002]
[glusterd-server-quorum.c:346:glusterd_do_volume_quorum_action]
0-management: Server quorum lost for volume data. Stopping local bricks.
[2017-05-22 18:36:24.740516] C [MSGID: 106002]
[glusterd-server-quorum.c:346:glusterd_do_volume_quorum_action]
0-management: Server quorum lost for volume engine. Stopping local bricks.
[2017-05-22 18:36:24.742215] C [MSGID: 106002]
[glusterd-server-quorum.c:346:glusterd_do_volume_quorum_action]
0-management: Server quorum lost for volume iso. Stoppin

Re: [Gluster-users] 120k context switches on GlsuterFS nodes

[Joe Julian]

On 05/22/17 10:27, mabi wrote:
Sorry for posting again but I was really wondering if it is somehow 
possible to tune gluster in order to make better use of all my cores 
(see below for the details). I suspect that is the reason for the high 
sporadic context switches I have been experiencing.


Cheers!



In theory, more clients and more diverse filesets.

The only way to know would be for you to analyze the traffic pattern 
and/or profile gluster on your server. There's never some magic "tune 
software X to operate more efficiently" setting, or else it would be the 
default (except for the "turbo" button back in the early PC clone days).






 Original Message 
Subject: Re: [Gluster-users] 120k context switches on GlsuterFS nodes
Local Time: May 18, 2017 8:43 PM
UTC Time: May 18, 2017 6:43 PM
From: m...@protonmail.ch
To: Ravishankar N 
Pranith Kumar Karampuri , Gluster Users 
, Gluster Devel 


I have a single Intel Xeon CPU E5-2620 v3 @ 2.40GHz in each nodes and 
this one has 6 cores and 12 threads. I thought this would be enough 
for GlusterFS. When I check my CPU graphs everything is pretty much 
idle and there is hardly any peeks at all on the CPU. During the very 
high context switch my CPU graphs shows the following:


1 thread was 100% busy in CPU user
1 thread was 100% busy in CPU system

leaving actually 10 other threads out of the total of 12 threads 
unused...


Is there maybe any performance tuning parameters I need to configure 
in order to make a better use of my CPU cores or threads?






 Original Message 
Subject: Re: [Gluster-users] 120k context switches on GlsuterFS nodes
Local Time: May 18, 2017 7:03 AM
UTC Time: May 18, 2017 5:03 AM
From: ravishan...@redhat.com
To: Pranith Kumar Karampuri , mabi 

Gluster Users , Gluster Devel 




On 05/17/2017 11:07 PM, Pranith Kumar Karampuri wrote:

+ gluster-devel

On Wed, May 17, 2017 at 10:50 PM, mabi > wrote:


I don't know exactly what kind of context-switches it was but
what I know is that it is the "cs" number under "system" when
you run vmstat.

Okay, that could be due to the  syscalls themselves or pre-emptive 
multitasking in case there aren't enough cpu cores. I think the 
spike in numbers is due to more users accessing the files at the 
same time like you observed, translating into more syscalls.  You 
can try capturing the gluster volume profile info the next time it 
occurs and co-relate with the cs count. If you don't see any 
negative performance impact, I think you don't need to be bothered 
much by the numbers.


HTH,
Ravi



Also I use the percona linux monitoring template for cacti

(https://www.percona.com/doc/percona-monitoring-plugins/LATEST/cacti/linux-templates.html

)
which monitors context switches too. If that's of any use
interrupts where also quite high during that time with peaks up
to 50k interrupts.




 Original Message 
Subject: Re: [Gluster-users] 120k context switches on
GlsuterFS nodes
Local Time: May 17, 2017 2:37 AM
UTC Time: May 17, 2017 12:37 AM
From: ravishan...@redhat.com 
To: mabi mailto:m...@protonmail.ch>>,
Gluster Users mailto:gluster-users@gluster.org>>


On 05/16/2017 11:13 PM, mabi wrote:

Today I even saw up to 400k context switches for around 30
minutes on my two nodes replica... Does anyone else have so
high context switches on their GlusterFS nodes?

I am wondering what is "normal" and if I should be worried...





 Original Message 
Subject: 120k context switches on GlsuterFS nodes
Local Time: May 11, 2017 9:18 PM
UTC Time: May 11, 2017 7:18 PM
From: m...@protonmail.ch 
To: Gluster Users 


Hi,

Today I noticed that for around 50 minutes my two GlusterFS
3.8.11 nodes had a very high amount of context switches,
around 120k. Usually the average is more around 1k-2k. So I
checked what was happening and there where just more users
accessing (downloading) their files at the same time. These
are directories with typical cloud files, which means files
of any sizes ranging from a few kB to MB and a lot of course.

Now I never saw such a high number in context switches in my
entire life so I wanted to ask if this is normal or to be
expected? I do not find any signs of errors or warnings in
any log files.



What context switch are you referring to (syscalls
context-switch on the bricks?) ? How did you measure this?
-Ravi


My volume is a replicated volume on two nodes with ZFS as
filesystem behind and the volume is mounted using FUSE on
the client (the cloud server). On that cloud server the
glusterfs process was using quite a lot of system 

Re: [Gluster-users] 120k context switches on GlsuterFS nodes

[mabi]

Sorry for posting again but I was really wondering if it is somehow possible to 
tune gluster in order to make better use of all my cores (see below for the 
details). I suspect that is the reason for the high sporadic context switches I 
have been experiencing.

Cheers!

 Original Message 
Subject: Re: [Gluster-users] 120k context switches on GlsuterFS nodes
Local Time: May 18, 2017 8:43 PM
UTC Time: May 18, 2017 6:43 PM
From: m...@protonmail.ch
To: Ravishankar N 
Pranith Kumar Karampuri , Gluster Users 
, Gluster Devel 

I have a single Intel Xeon CPU E5-2620 v3 @ 2.40GHz in each nodes and this one 
has 6 cores and 12 threads. I thought this would be enough for GlusterFS. When 
I check my CPU graphs everything is pretty much idle and there is hardly any 
peeks at all on the CPU. During the very high context switch my CPU graphs 
shows the following:

1 thread was 100% busy in CPU user
1 thread was 100% busy in CPU system

leaving actually 10 other threads out of the total of 12 threads unused...

Is there maybe any performance tuning parameters I need to configure in order 
to make a better use of my CPU cores or threads?

 Original Message 
Subject: Re: [Gluster-users] 120k context switches on GlsuterFS nodes
Local Time: May 18, 2017 7:03 AM
UTC Time: May 18, 2017 5:03 AM
From: ravishan...@redhat.com
To: Pranith Kumar Karampuri , mabi 
Gluster Users , Gluster Devel 


On 05/17/2017 11:07 PM, Pranith Kumar Karampuri wrote:
+ gluster-devel

On Wed, May 17, 2017 at 10:50 PM, mabi  wrote:

I don't know exactly what kind of context-switches it was but what I know is 
that it is the "cs" number under "system" when you run vmstat.
Okay, that could be due to the syscalls themselves or pre-emptive multitasking 
in case there aren't enough cpu cores. I think the spike in numbers is due to 
more users accessing the files at the same time like you observed, translating 
into more syscalls. You can try capturing the gluster volume profile info the 
next time it occurs and co-relate with the cs count. If you don't see any 
negative performance impact, I think you don't need to be bothered much by the 
numbers.

HTH,
Ravi

Also I use the percona linux monitoring template for cacti 
(https://www.percona.com/doc/percona-monitoring-plugins/LATEST/cacti/linux-templates.html)
 which monitors context switches too. If that's of any use interrupts where 
also quite high during that time with peaks up to 50k interrupts.

 Original Message 
Subject: Re: [Gluster-users] 120k context switches on GlsuterFS nodes
Local Time: May 17, 2017 2:37 AM
UTC Time: May 17, 2017 12:37 AM
From: ravishan...@redhat.com
To: mabi , Gluster Users 

On 05/16/2017 11:13 PM, mabi wrote:
Today I even saw up to 400k context switches for around 30 minutes on my two 
nodes replica... Does anyone else have so high context switches on their 
GlusterFS nodes?

I am wondering what is "normal" and if I should be worried...

 Original Message 
Subject: 120k context switches on GlsuterFS nodes
Local Time: May 11, 2017 9:18 PM
UTC Time: May 11, 2017 7:18 PM
From: m...@protonmail.ch
To: Gluster Users 
[](mailto:gluster-users@gluster.org)

Hi,

Today I noticed that for around 50 minutes my two GlusterFS 3.8.11 nodes had a 
very high amount of context switches, around 120k. Usually the average is more 
around 1k-2k. So I checked what was happening and there where just more users 
accessing (downloading) their files at the same time. These are directories 
with typical cloud files, which means files of any sizes ranging from a few kB 
to MB and a lot of course.

Now I never saw such a high number in context switches in my entire life so I 
wanted to ask if this is normal or to be expected? I do not find any signs of 
errors or warnings in any log files.

What context switch are you referring to (syscalls context-switch on the 
bricks?) ? How did you measure this?
-Ravi

My volume is a replicated volume on two nodes with ZFS as filesystem behind and 
the volume is mounted using FUSE on the client (the cloud server). On that 
cloud server the glusterfs process was using quite a lot of system CPU but that 
server (VM) only has 2 vCPUs so maybe I should increase the number of vCPUs...

Any ideas or recommendations?

Regards,
M.

__

_
Gluster-users mailing list
Gluster-users@gluster.org

[http://lists.gluster.org/

mailman/listinfo/gluster-users](http://lists.gluster.org/mailman/listinfo/gluster-users)

___ Gluster-users mailing list 
Gluster-users@gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-users
--

Pranith___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failure while upgrading gluster to 3.10.1

[Atin Mukherjee]

On Mon, May 22, 2017 at 9:05 PM, Pawan Alwandi  wrote:

>
> On Mon, May 22, 2017 at 8:36 PM, Atin Mukherjee 
> wrote:
>
>>
>>
>> On Mon, May 22, 2017 at 7:51 PM, Atin Mukherjee 
>> wrote:
>>
>>> Sorry Pawan, I did miss the other part of the attachments. So looking
>>> from the glusterd.info file from all the hosts, it looks like host2 and
>>> host3 do not have the correct op-version. Can you please set the op-version
>>> as "operating-version=30702" in host2 and host3 and restart glusterd
>>> instance one by one on all the nodes?
>>>
>>
>> Please ensure that all the hosts are upgraded to the same bits before
>> doing this change.
>>
>
> Having to upgrade all 3 hosts to newer version before gluster could work
> successfully on any of them means application downtime.  The applications
> running on these hosts are expected to be highly available.  So with the
> way the things are right now, is an online upgrade possible?  My upgrade
> steps are: (1) stop the applications (2) umount the gluster volume, and
> then (3) upgrade gluster one host at a time.
>

One of the way to mitigate this is to first do an online upgrade to
glusterfs-3.7.9 (op-version:30707) given this bug was introduced in 3.7.10
and then come to 3.11.


> Our goal is to get gluster upgraded to 3.11 from 3.6.9, and to make this
> an online upgrade we are okay to take two steps 3.6.9 -> 3.7 and then 3.7
> to 3.11.
>
>
>>
>>
>>>
>>> Apparently it looks like there is a bug which you have uncovered, during
>>> peer handshaking if one of the glusterd instance is running with old bits
>>> then during validating the handshake request there is a possibility that
>>> uuid received will be blank and the same was ignored however there was a
>>> patch http://review.gluster.org/13519 which had some additional changes
>>> which was always looking at this field and doing some extra checks which
>>> was causing the handshake to fail. For now, the above workaround should
>>> suffice. I'll be sending a patch pretty soon.
>>>
>>
>> Posted a patch https://review.gluster.org/#/c/17358 .
>>
>>
>>>
>>>
>>>
>>> On Mon, May 22, 2017 at 11:35 AM, Pawan Alwandi 
>>> wrote:
>>>
 Hello Atin,

 The tar's have the content of `/var/lib/glusterd` too for all 3 nodes,
 please check again.

 Thanks

 On Mon, May 22, 2017 at 11:32 AM, Atin Mukherjee 
 wrote:

> Pawan,
>
> I see you have provided the log files from the nodes, however it'd be
> really helpful if you can provide me the content of /var/lib/glusterd from
> all the nodes to get to the root cause of this issue.
>
> On Fri, May 19, 2017 at 12:09 PM, Pawan Alwandi 
> wrote:
>
>> Hello Atin,
>>
>> Thanks for continued support.  I've attached requested files from all
>> 3 nodes.
>>
>> (I think we already verified the UUIDs to be correct, anyway let us
>> know if you find any more info in the logs)
>>
>> Pawan
>>
>> On Thu, May 18, 2017 at 11:45 PM, Atin Mukherjee > > wrote:
>>
>>>
>>> On Thu, 18 May 2017 at 23:40, Atin Mukherjee 
>>> wrote:
>>>
 On Wed, 17 May 2017 at 12:47, Pawan Alwandi 
 wrote:

> Hello Atin,
>
> I realized that these http://gluster.readthedocs.io/
> en/latest/Upgrade-Guide/upgrade_to_3.10/ instructions only work
> for upgrades from 3.7, while we are running 3.6.2.  Are there
> instructions/suggestion you have for us to upgrade from 3.6 version?
>
> I believe upgrade from 3.6 to 3.7 and then to 3.10 would work, but
> I see similar errors reported when I upgraded to 3.7 too.
>
> For what its worth, I was able to set the op-version (gluster v
> set all cluster.op-version 30702) but that doesn't seem to help.
>
> [2017-05-17 06:48:33.700014] I [MSGID: 100030]
> [glusterfsd.c:2338:main] 0-/usr/sbin/glusterd: Started running
> /usr/sbin/glusterd version 3.7.20 (args: /usr/sbin/glusterd -p
> /var/run/glusterd.pid)
> [2017-05-17 06:48:33.703808] I [MSGID: 106478]
> [glusterd.c:1383:init] 0-management: Maximum allowed open file 
> descriptors
> set to 65536
> [2017-05-17 06:48:33.703836] I [MSGID: 106479]
> [glusterd.c:1432:init] 0-management: Using /var/lib/glusterd as 
> working
> directory
> [2017-05-17 06:48:33.708866] W [MSGID: 103071]
> [rdma.c:4594:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm
> event channel creation failed [No such device]
> [2017-05-17 06:48:33.709011] W [MSGID: 103055] [rdma.c:4901:init]
> 0-rdma.management: Failed to initialize IB Device
> [2017-05-17 06:48:33.709033] W 
> [rpc-transport.c:359:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
> [2017-05-17 06:48:33.709088] W [rpcsvc.c:1642:rpcsvc_create_listener]
> 0-rpc

Re: [Gluster-users] GlusterFS and Kafka

[Joe Julian]

This may be asking too much, but can you explain why or how it's even possible 
to bypass the cache like this, Vijay?


On May 22, 2017 7:41:40 AM PDT, Vijay Bellur  wrote:
>Looks like a problem with caching. Can you please try by disabling all
>performance translators? The following configuration commands would
>disable
>performance translators in the gluster client stack:
>
>gluster volume set  performance.quick-read off
>gluster volume set  performance.io-cache off
>gluster volume set  performance.write-behind off
>gluster volume set  performance.stat-prefetch off
>gluster volume set  performance.read-ahead off
>gluster volume set  performance.readdir-ahead off
>gluster volume set  performance.open-behind off
>gluster volume set  performance.client-io-threads off
>
>Thanks,
>Vijay
>
>
>
>On Mon, May 22, 2017 at 9:46 AM, Christopher Schmidt
>
>wrote:
>
>> Hi all,
>>
>> has anyone ever successfully deployed a Kafka (Cluster) on GlusterFS
>> volumes?
>>
>> I my case it's a Kafka Kubernetes-StatefulSet and a Heketi GlusterFS.
>> Needless to say that I am getting a lot of filesystem related
>exceptions
>> like this one:
>>
>> Failed to read `log header` from file channel
>`sun.nio.ch.FileChannelImpl@67afa54a`.
>> Expected to read 12 bytes, but reached end of file after reading 0
>bytes.
>> Started read from position 123065680.
>>
>> I limited the amount of exceptions with the
>log.flush.interval.messages=1
>> option, but not all...
>>
>> best Christopher
>>
>>
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failure while upgrading gluster to 3.10.1

[Pawan Alwandi]

On Mon, May 22, 2017 at 8:36 PM, Atin Mukherjee  wrote:

>
>
> On Mon, May 22, 2017 at 7:51 PM, Atin Mukherjee 
> wrote:
>
>> Sorry Pawan, I did miss the other part of the attachments. So looking
>> from the glusterd.info file from all the hosts, it looks like host2 and
>> host3 do not have the correct op-version. Can you please set the op-version
>> as "operating-version=30702" in host2 and host3 and restart glusterd
>> instance one by one on all the nodes?
>>
>
> Please ensure that all the hosts are upgraded to the same bits before
> doing this change.
>

Having to upgrade all 3 hosts to newer version before gluster could work
successfully on any of them means application downtime.  The applications
running on these hosts are expected to be highly available.  So with the
way the things are right now, is an online upgrade possible?  My upgrade
steps are: (1) stop the applications (2) umount the gluster volume, and
then (3) upgrade gluster one host at a time.

Our goal is to get gluster upgraded to 3.11 from 3.6.9, and to make this an
online upgrade we are okay to take two steps 3.6.9 -> 3.7 and then 3.7 to
3.11.


>
>
>>
>> Apparently it looks like there is a bug which you have uncovered, during
>> peer handshaking if one of the glusterd instance is running with old bits
>> then during validating the handshake request there is a possibility that
>> uuid received will be blank and the same was ignored however there was a
>> patch http://review.gluster.org/13519 which had some additional changes
>> which was always looking at this field and doing some extra checks which
>> was causing the handshake to fail. For now, the above workaround should
>> suffice. I'll be sending a patch pretty soon.
>>
>
> Posted a patch https://review.gluster.org/#/c/17358 .
>
>
>>
>>
>>
>> On Mon, May 22, 2017 at 11:35 AM, Pawan Alwandi 
>> wrote:
>>
>>> Hello Atin,
>>>
>>> The tar's have the content of `/var/lib/glusterd` too for all 3 nodes,
>>> please check again.
>>>
>>> Thanks
>>>
>>> On Mon, May 22, 2017 at 11:32 AM, Atin Mukherjee 
>>> wrote:
>>>
 Pawan,

 I see you have provided the log files from the nodes, however it'd be
 really helpful if you can provide me the content of /var/lib/glusterd from
 all the nodes to get to the root cause of this issue.

 On Fri, May 19, 2017 at 12:09 PM, Pawan Alwandi 
 wrote:

> Hello Atin,
>
> Thanks for continued support.  I've attached requested files from all
> 3 nodes.
>
> (I think we already verified the UUIDs to be correct, anyway let us
> know if you find any more info in the logs)
>
> Pawan
>
> On Thu, May 18, 2017 at 11:45 PM, Atin Mukherjee 
> wrote:
>
>>
>> On Thu, 18 May 2017 at 23:40, Atin Mukherjee 
>> wrote:
>>
>>> On Wed, 17 May 2017 at 12:47, Pawan Alwandi 
>>> wrote:
>>>
 Hello Atin,

 I realized that these http://gluster.readthedocs.io/
 en/latest/Upgrade-Guide/upgrade_to_3.10/ instructions only work
 for upgrades from 3.7, while we are running 3.6.2.  Are there
 instructions/suggestion you have for us to upgrade from 3.6 version?

 I believe upgrade from 3.6 to 3.7 and then to 3.10 would work, but
 I see similar errors reported when I upgraded to 3.7 too.

 For what its worth, I was able to set the op-version (gluster v set
 all cluster.op-version 30702) but that doesn't seem to help.

 [2017-05-17 06:48:33.700014] I [MSGID: 100030]
 [glusterfsd.c:2338:main] 0-/usr/sbin/glusterd: Started running
 /usr/sbin/glusterd version 3.7.20 (args: /usr/sbin/glusterd -p
 /var/run/glusterd.pid)
 [2017-05-17 06:48:33.703808] I [MSGID: 106478]
 [glusterd.c:1383:init] 0-management: Maximum allowed open file 
 descriptors
 set to 65536
 [2017-05-17 06:48:33.703836] I [MSGID: 106479]
 [glusterd.c:1432:init] 0-management: Using /var/lib/glusterd as working
 directory
 [2017-05-17 06:48:33.708866] W [MSGID: 103071]
 [rdma.c:4594:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm
 event channel creation failed [No such device]
 [2017-05-17 06:48:33.709011] W [MSGID: 103055] [rdma.c:4901:init]
 0-rdma.management: Failed to initialize IB Device
 [2017-05-17 06:48:33.709033] W [rpc-transport.c:359:rpc_transport_load]
 0-rpc-transport: 'rdma' initialization failed
 [2017-05-17 06:48:33.709088] W [rpcsvc.c:1642:rpcsvc_create_listener]
 0-rpc-service: cannot create listener, initing the transport failed
 [2017-05-17 06:48:33.709105] E [MSGID: 106243]
 [glusterd.c:1656:init] 0-management: creation of 1 listeners failed,
 continuing with succeeded transport
 [2017-05-17 06:48:35.480043] I [MSGID: 106513]
 [glusterd-store.c:2068:glusterd_restore_op_version] 0-glusterd:
 retri

Re: [Gluster-users] Failure while upgrading gluster to 3.10.1

[Atin Mukherjee]

On Mon, May 22, 2017 at 7:51 PM, Atin Mukherjee  wrote:

> Sorry Pawan, I did miss the other part of the attachments. So looking from
> the glusterd.info file from all the hosts, it looks like host2 and host3
> do not have the correct op-version. Can you please set the op-version as
> "operating-version=30702" in host2 and host3 and restart glusterd instance
> one by one on all the nodes?
>

Please ensure that all the hosts are upgraded to the same bits before doing
this change.


>
> Apparently it looks like there is a bug which you have uncovered, during
> peer handshaking if one of the glusterd instance is running with old bits
> then during validating the handshake request there is a possibility that
> uuid received will be blank and the same was ignored however there was a
> patch http://review.gluster.org/13519 which had some additional changes
> which was always looking at this field and doing some extra checks which
> was causing the handshake to fail. For now, the above workaround should
> suffice. I'll be sending a patch pretty soon.
>

Posted a patch https://review.gluster.org/#/c/17358 .


>
>
>
> On Mon, May 22, 2017 at 11:35 AM, Pawan Alwandi  wrote:
>
>> Hello Atin,
>>
>> The tar's have the content of `/var/lib/glusterd` too for all 3 nodes,
>> please check again.
>>
>> Thanks
>>
>> On Mon, May 22, 2017 at 11:32 AM, Atin Mukherjee 
>> wrote:
>>
>>> Pawan,
>>>
>>> I see you have provided the log files from the nodes, however it'd be
>>> really helpful if you can provide me the content of /var/lib/glusterd from
>>> all the nodes to get to the root cause of this issue.
>>>
>>> On Fri, May 19, 2017 at 12:09 PM, Pawan Alwandi 
>>> wrote:
>>>
 Hello Atin,

 Thanks for continued support.  I've attached requested files from all 3
 nodes.

 (I think we already verified the UUIDs to be correct, anyway let us
 know if you find any more info in the logs)

 Pawan

 On Thu, May 18, 2017 at 11:45 PM, Atin Mukherjee 
 wrote:

>
> On Thu, 18 May 2017 at 23:40, Atin Mukherjee 
> wrote:
>
>> On Wed, 17 May 2017 at 12:47, Pawan Alwandi 
>> wrote:
>>
>>> Hello Atin,
>>>
>>> I realized that these http://gluster.readthedocs.io/
>>> en/latest/Upgrade-Guide/upgrade_to_3.10/ instructions only work for
>>> upgrades from 3.7, while we are running 3.6.2.  Are there
>>> instructions/suggestion you have for us to upgrade from 3.6 version?
>>>
>>> I believe upgrade from 3.6 to 3.7 and then to 3.10 would work, but I
>>> see similar errors reported when I upgraded to 3.7 too.
>>>
>>> For what its worth, I was able to set the op-version (gluster v set
>>> all cluster.op-version 30702) but that doesn't seem to help.
>>>
>>> [2017-05-17 06:48:33.700014] I [MSGID: 100030]
>>> [glusterfsd.c:2338:main] 0-/usr/sbin/glusterd: Started running
>>> /usr/sbin/glusterd version 3.7.20 (args: /usr/sbin/glusterd -p
>>> /var/run/glusterd.pid)
>>> [2017-05-17 06:48:33.703808] I [MSGID: 106478]
>>> [glusterd.c:1383:init] 0-management: Maximum allowed open file 
>>> descriptors
>>> set to 65536
>>> [2017-05-17 06:48:33.703836] I [MSGID: 106479]
>>> [glusterd.c:1432:init] 0-management: Using /var/lib/glusterd as working
>>> directory
>>> [2017-05-17 06:48:33.708866] W [MSGID: 103071]
>>> [rdma.c:4594:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm
>>> event channel creation failed [No such device]
>>> [2017-05-17 06:48:33.709011] W [MSGID: 103055] [rdma.c:4901:init]
>>> 0-rdma.management: Failed to initialize IB Device
>>> [2017-05-17 06:48:33.709033] W [rpc-transport.c:359:rpc_transport_load]
>>> 0-rpc-transport: 'rdma' initialization failed
>>> [2017-05-17 06:48:33.709088] W [rpcsvc.c:1642:rpcsvc_create_listener]
>>> 0-rpc-service: cannot create listener, initing the transport failed
>>> [2017-05-17 06:48:33.709105] E [MSGID: 106243]
>>> [glusterd.c:1656:init] 0-management: creation of 1 listeners failed,
>>> continuing with succeeded transport
>>> [2017-05-17 06:48:35.480043] I [MSGID: 106513]
>>> [glusterd-store.c:2068:glusterd_restore_op_version] 0-glusterd:
>>> retrieved op-version: 30600
>>> [2017-05-17 06:48:35.605779] I [MSGID: 106498]
>>> [glusterd-handler.c:3640:glusterd_friend_add_from_peerinfo]
>>> 0-management: connect returned 0
>>> [2017-05-17 06:48:35.607059] I 
>>> [rpc-clnt.c:1046:rpc_clnt_connection_init]
>>> 0-management: setting frame-timeout to 600
>>> [2017-05-17 06:48:35.607670] I 
>>> [rpc-clnt.c:1046:rpc_clnt_connection_init]
>>> 0-management: setting frame-timeout to 600
>>> [2017-05-17 06:48:35.607025] I [MSGID: 106498]
>>> [glusterd-handler.c:3640:glusterd_friend_add_from_peerinfo]
>>> 0-management: connect returned 0
>>> [2017-05-17 06:48:35.608125] I [MSGID: 106544]
>>> [glusterd.c:159:glusterd_uuid_init] 0-management: retr

Re: [Gluster-users] GlusterFS and Kafka

[Vijay Bellur]

Looks like a problem with caching. Can you please try by disabling all
performance translators? The following configuration commands would disable
performance translators in the gluster client stack:

gluster volume set  performance.quick-read off
gluster volume set  performance.io-cache off
gluster volume set  performance.write-behind off
gluster volume set  performance.stat-prefetch off
gluster volume set  performance.read-ahead off
gluster volume set  performance.readdir-ahead off
gluster volume set  performance.open-behind off
gluster volume set  performance.client-io-threads off

Thanks,
Vijay



On Mon, May 22, 2017 at 9:46 AM, Christopher Schmidt 
wrote:

> Hi all,
>
> has anyone ever successfully deployed a Kafka (Cluster) on GlusterFS
> volumes?
>
> I my case it's a Kafka Kubernetes-StatefulSet and a Heketi GlusterFS.
> Needless to say that I am getting a lot of filesystem related exceptions
> like this one:
>
> Failed to read `log header` from file channel 
> `sun.nio.ch.FileChannelImpl@67afa54a`.
> Expected to read 12 bytes, but reached end of file after reading 0 bytes.
> Started read from position 123065680.
>
> I limited the amount of exceptions with the log.flush.interval.messages=1
> option, but not all...
>
> best Christopher
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Failure while upgrading gluster to 3.10.1

[Atin Mukherjee]

Sorry Pawan, I did miss the other part of the attachments. So looking from
the glusterd.info file from all the hosts, it looks like host2 and host3 do
not have the correct op-version. Can you please set the op-version as
"operating-version=30702" in host2 and host3 and restart glusterd instance
one by one on all the nodes?

Apparently it looks like there is a bug which you have uncovered, during
peer handshaking if one of the glusterd instance is running with old bits
then during validating the handshake request there is a possibility that
uuid received will be blank and the same was ignored however there was a
patch http://review.gluster.org/13519 which had some additional changes
which was always looking at this field and doing some extra checks which
was causing the handshake to fail. For now, the above workaround should
suffice. I'll be sending a patch pretty soon.




On Mon, May 22, 2017 at 11:35 AM, Pawan Alwandi  wrote:

> Hello Atin,
>
> The tar's have the content of `/var/lib/glusterd` too for all 3 nodes,
> please check again.
>
> Thanks
>
> On Mon, May 22, 2017 at 11:32 AM, Atin Mukherjee 
> wrote:
>
>> Pawan,
>>
>> I see you have provided the log files from the nodes, however it'd be
>> really helpful if you can provide me the content of /var/lib/glusterd from
>> all the nodes to get to the root cause of this issue.
>>
>> On Fri, May 19, 2017 at 12:09 PM, Pawan Alwandi 
>> wrote:
>>
>>> Hello Atin,
>>>
>>> Thanks for continued support.  I've attached requested files from all 3
>>> nodes.
>>>
>>> (I think we already verified the UUIDs to be correct, anyway let us know
>>> if you find any more info in the logs)
>>>
>>> Pawan
>>>
>>> On Thu, May 18, 2017 at 11:45 PM, Atin Mukherjee 
>>> wrote:
>>>

 On Thu, 18 May 2017 at 23:40, Atin Mukherjee 
 wrote:

> On Wed, 17 May 2017 at 12:47, Pawan Alwandi  wrote:
>
>> Hello Atin,
>>
>> I realized that these http://gluster.readthedocs.io/
>> en/latest/Upgrade-Guide/upgrade_to_3.10/ instructions only work for
>> upgrades from 3.7, while we are running 3.6.2.  Are there
>> instructions/suggestion you have for us to upgrade from 3.6 version?
>>
>> I believe upgrade from 3.6 to 3.7 and then to 3.10 would work, but I
>> see similar errors reported when I upgraded to 3.7 too.
>>
>> For what its worth, I was able to set the op-version (gluster v set
>> all cluster.op-version 30702) but that doesn't seem to help.
>>
>> [2017-05-17 06:48:33.700014] I [MSGID: 100030]
>> [glusterfsd.c:2338:main] 0-/usr/sbin/glusterd: Started running
>> /usr/sbin/glusterd version 3.7.20 (args: /usr/sbin/glusterd -p
>> /var/run/glusterd.pid)
>> [2017-05-17 06:48:33.703808] I [MSGID: 106478] [glusterd.c:1383:init]
>> 0-management: Maximum allowed open file descriptors set to 65536
>> [2017-05-17 06:48:33.703836] I [MSGID: 106479] [glusterd.c:1432:init]
>> 0-management: Using /var/lib/glusterd as working directory
>> [2017-05-17 06:48:33.708866] W [MSGID: 103071]
>> [rdma.c:4594:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm
>> event channel creation failed [No such device]
>> [2017-05-17 06:48:33.709011] W [MSGID: 103055] [rdma.c:4901:init]
>> 0-rdma.management: Failed to initialize IB Device
>> [2017-05-17 06:48:33.709033] W [rpc-transport.c:359:rpc_transport_load]
>> 0-rpc-transport: 'rdma' initialization failed
>> [2017-05-17 06:48:33.709088] W [rpcsvc.c:1642:rpcsvc_create_listener]
>> 0-rpc-service: cannot create listener, initing the transport failed
>> [2017-05-17 06:48:33.709105] E [MSGID: 106243] [glusterd.c:1656:init]
>> 0-management: creation of 1 listeners failed, continuing with succeeded
>> transport
>> [2017-05-17 06:48:35.480043] I [MSGID: 106513]
>> [glusterd-store.c:2068:glusterd_restore_op_version] 0-glusterd:
>> retrieved op-version: 30600
>> [2017-05-17 06:48:35.605779] I [MSGID: 106498]
>> [glusterd-handler.c:3640:glusterd_friend_add_from_peerinfo]
>> 0-management: connect returned 0
>> [2017-05-17 06:48:35.607059] I [rpc-clnt.c:1046:rpc_clnt_connection_init]
>> 0-management: setting frame-timeout to 600
>> [2017-05-17 06:48:35.607670] I [rpc-clnt.c:1046:rpc_clnt_connection_init]
>> 0-management: setting frame-timeout to 600
>> [2017-05-17 06:48:35.607025] I [MSGID: 106498]
>> [glusterd-handler.c:3640:glusterd_friend_add_from_peerinfo]
>> 0-management: connect returned 0
>> [2017-05-17 06:48:35.608125] I [MSGID: 106544]
>> [glusterd.c:159:glusterd_uuid_init] 0-management: retrieved UUID:
>> 7f2a6e11-2a53-4ab4-9ceb-8be6a9f2d073
>>
>
>> Final graph:
>> +---
>> ---+
>>   1: volume management
>>   2: type mgmt/glusterd
>>   3: option rpc-auth.auth-glusterfs on
>>   4: option rpc-auth.auth-unix on
>>   5: option rpc-auth.a

[Gluster-users] gluster server daemon connection refused

[Richard Neuboeck]

Hi Group,

I'm having some difficulty debugging a problem (or maybe more than
one problem) I have a replica 3 gluster file system. Basic setup is
three machines, all CentOS 7 (up to date), gluster 3.8 (provided by
the CentOS repos). XFS brick partitions.

At a (so far random) point in time the gluster server daemons stop
communicating with each other. The glusterfs processes for each
volume on each server are still running though. If I restart them
manually (systemctl restart glusterfsd glusterd) communication is
once again established.

(For me) there is no obvious log message that would point me to the
source of the problem.

In general all servers show a lot of different warnings and errors
in the brick logs almost all the time clients are connected. All
clients are Fedora 25 machines using the FUSE gluster client (3.8)

From the brick log on the storage servers:
It seems the quota sub system has some problems:
[2017-05-22 11:41:36.323233] W [marker-quota.c:33:mq_loc_copy]
0-marker: src loc is not valid
[2017-05-22 11:41:36.323269] E
[marker-quota.c:1472:mq_initiate_quota_task] 0-home-marker: loc copy
failed

And there are a lot of warnings about xattrs like this:
[2017-05-22 11:32:28.111240] W [MSGID: 113001]
[posix.c:4212:posix_get_ancestry_non_directory] 0-home-posix:
listxattr failed
on/srv/gluster_home/brick/.glusterfs/34/06/3406a026-596b-40b3-8dd1-8186e3072031
[No such file or directory]

And I get some of these:
[2017-05-22 11:41:29.591258] E [MSGID: 115081]
[server-rpc-fops.c:1201:server_fstat_cbk] 0-home-server: 923966:
FSTAT -2 (197f90db-ff62-4436-b200-f4347b6c2ba0) ==> (Operation not
permitted) [Operation not permitted]



There are some warnings in the glustershd.log but none prior to the
disconnect give me a clue as to what's going on:

[2017-05-22 07:37:48.003835] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-home-client-1:
remote operation failed. Path:

(b8812b00-7bda-4871-ab39-82d1eb8a2607) [No such file or directory]
[2017-05-22 07:37:48.003842] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-home-client-0:
remote operation failed. Path:

(b8812b00-7bda-4871-ab39-82d1eb8a2607) [No such file or directory]
[2017-05-22 07:37:48.003896] W [MSGID: 114031]
[client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-home-client-2:
remote operation failed. Path:

(b8812b00-7bda-4871-ab39-82d1eb8a2607) [No such file or directory]
[2017-05-22 11:14:36.832051] W [socket.c:590:__socket_rwv]
0-home-client-1: readv on X.X.X.8:49156 failed (No data available)
[2017-05-22 11:14:36.832103] I [MSGID: 114018]
[client.c:2280:client_rpc_notify] 0-home-client-1: disconnected from
home-client-1. Client process will keep trying to connect to
glusterd until brick's port is available
[2017-05-22 11:14:36.832295] W [socket.c:590:__socket_rwv]
0-home-client-2: readv on X.X.X.9:49156 failed (No data available)
[2017-05-22 11:14:36.832317] I [MSGID: 114018]
[client.c:2280:client_rpc_notify] 0-home-client-2: disconnected from
home-client-2. Client process will keep trying to connect to
glusterd until brick's port is available
[2017-05-22 11:14:36.832328] W [MSGID: 108001]
[afr-common.c:4467:afr_notify] 0-home-replicate-0: Client-quorum is
not met
[2017-05-22 11:14:47.423671] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-home-client-1: changing port to 49156 (from 0)
[2017-05-22 11:14:47.430133] I [rpc-clnt.c:1965:rpc_clnt_reconfig]
0-home-client-2: changing port to 49156 (from 0)
[2017-05-22 11:14:47.434348] E [socket.c:2309:socket_connect_finish]
0-home-client-1: connection to X.X.X.8:49156 failed (Connection refused)
[2017-05-22 11:14:47.438925] E [socket.c:2309:socket_connect_finish]
0-home-client-2: connection to X.X.X.9:49156 failed (Connection refused)


Gluster Volume Info:
Volume Name: home
Type: Replicate
Volume ID: 47875706-1ed1-48b7-bc5d-0dca8ec5cd58
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: four:/srv/gluster_home/brick
Brick2: five:/srv/gluster_home/brick
Brick3: six:/srv/gluster_home/brick
Options Reconfigured:
server.root-squash: off
network.ping-timeout: 10
cluster.quorum-type: auto
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
features.quota: on
features.inode-quota: on
features.quota-deem-statfs: on

Installed gluster server (CentOS 7) version:
glusterfs-server-3.8.11-1.el7.x86_64

Installed gluster client (Fedora 25) version:
glusterfs-3.8.9-1.fc24.x86_64


Does anybody have an idea why the communication might be interrupted?

And also why there a so many warnings about extended attributes of
non existing files?

I'm running basically the same setup with CentOS clients for oVirt
without any gluster warnings or errors at all. Currently I'm at a
loss. Any help is highly appreciated!
Thanks
Richard Neuboeck

-- 
-
[a] Department for Theoretical Chemistry
University of Vienna
Waehringer Strasse 17/3/304, 1090 Wien, Austria

[Gluster-users] GlusterFS and Kafka

[Christopher Schmidt]

Hi all,

has anyone ever successfully deployed a Kafka (Cluster) on GlusterFS
volumes?

I my case it's a Kafka Kubernetes-StatefulSet and a Heketi GlusterFS.
Needless to say that I am getting a lot of filesystem related exceptions
like this one:

Failed to read `log header` from file channel
`sun.nio.ch.FileChannelImpl@67afa54a`. Expected to read 12 bytes, but
reached end of file after reading 0 bytes. Started read from position
123065680.

I limited the amount of exceptions with the log.flush.interval.messages=1
option, but not all...

best Christopher
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Reasons for recommending nfs-ganesha

[Joe Julian]

On May 22, 2017 5:20:05 AM PDT, "Kaleb S. KEITHLEY"  wrote:
>On 05/19/2017 08:57 AM, te-yamau...@usen.co.jp wrote:
>> I currently use version 3.10.2.
>> When nfs is enabled, the following warning is displayed.
>> Why is nfs-ganesha recommended?
>> Is there something wrong with gluster nfs?
>> 
>
>This is hardly new. The migration to NFS-Ganesha started with 3.8.0 and
>there has been discussion about it on the Gluster community mailing
>lists over the past two years.
>
>gluster nfs (or gnfs) only supports NFSv3. The gnfs part doesn't have
>High Availability (HA) out of the box. The design of gnfs doesn't lend
>itself to adding NFSv4 and beyond.
>
>NFS-Ganesha has NFSv3, NFSv4, NFSv4.1, NFSv4.2, pNFS — and if you care
>about it, 9P support. It's a better implementation, and it's being
>actively developed and maintained.
>
>Besides kernel NFS (knfs) it doesn't make sense to pay for maintaining
>two very different NFS implementations. (IOW Red Hat isn't going to pay
>to maintain and develop two different implementations. Someone else may
>step up and maintain gnfs.)
>
>You can keep using gnfs, but eventually you should switch to
>NFS-Ganesha
>because that's where resources are devoted — for fixing bugs and adding
>features.
>
>You *should* switch, but nobody is going to force you to switch. The
>gnfs part will always be there and you'll always be able to use it.

To be accurate, gnfs is disabled by default now so you would have to enable it 
for your volume.

>
>-- 
>
>Kaleb
>___
>Gluster-users mailing list
>Gluster-users@gluster.org
>http://lists.gluster.org/mailman/listinfo/gluster-users

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Rebalance + VM corruption - current status and request for feedback

[Niels de Vos]

On Sun, May 21, 2017 at 02:37:34AM +, Atin Mukherjee wrote:
> On Sun, 21 May 2017 at 02:17, Mahdi Adnan  wrote:
> 
> > Thank you so much for your replay.
> >
> > Yes, i checked the 310 repository before and it was't there, see below;
> >
> 
> I think 3.10.2 bits are still in 3.10 test repository in storage sig and
> hasn't been pushed to release mirror yet. Niels can update further on this.

Nobody reported test results, so there was no urgency to push the
release to the CentOS mirrors. I've marked the packages for release now,
and the CentOS team will probably sign+mirror them during the day
tomorrow.

Niels


> 
> 
> > Installed Packages
> > Name: glusterfs-server
> > Arch: x86_64
> > Version : 3.10.1
> > Release : 1.el7
> > Size: 4.3 M
> > Repo: installed
> > From repo   : centos-gluster310
> > Summary : Distributed file-system server
> > URL : http://gluster.readthedocs.io/en/latest/
> > License : GPLv2 or LGPLv3+
> > Description :
> >
> > and no other packages available in the repo
> >
> > --
> >
> > Respectfully
> > *Mahdi A. Mahdi*
> >
> > --
> > *From:* Vijay Bellur 
> > *Sent:* Saturday, May 20, 2017 6:46:51 PM
> > *To:* Krutika Dhananjay
> > *Cc:* Mahdi Adnan; raghavendra talur; gluster-user
> > *Subject:* Re: [Gluster-users] Rebalance + VM corruption - current status
> > and request for feedback
> >
> >
> >
> > On Sat, May 20, 2017 at 6:38 AM, Krutika Dhananjay 
> > wrote:
> >
> >> Raghavendra Talur might know. Adding him to the thread.
> >>
> >> -Krutika
> >>
> >> On Sat, May 20, 2017 at 2:47 PM, Mahdi Adnan 
> >> wrote:
> >>
> >>> Good morning,
> >>>
> >>>
> >>> SIG repository does not have the latest glusterfs 3.10.2.
> >>>
> >>> Do you have any idea when it's going to be updated ?
> >>>
> >>> Is there any other recommended place to get the latest rpms ?
> >>>
> >>>
> >
> > RPMs are available in the centos-gluster310-test repository [1].
> >
> > -Vijay
> >
> > [1] http://lists.gluster.org/pipermail/maintainers/2017-May/002575.html
> >
> >
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users
> 
> -- 
> - Atin (atinm)


signature.asc
Description: PGP signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Reasons for recommending nfs-ganesha

[Kaleb S. KEITHLEY]

On 05/19/2017 08:57 AM, te-yamau...@usen.co.jp wrote:
> I currently use version 3.10.2.
> When nfs is enabled, the following warning is displayed.
> Why is nfs-ganesha recommended?
> Is there something wrong with gluster nfs?
> 

This is hardly new. The migration to NFS-Ganesha started with 3.8.0 and
there has been discussion about it on the Gluster community mailing
lists over the past two years.

gluster nfs (or gnfs) only supports NFSv3. The gnfs part doesn't have
High Availability (HA) out of the box. The design of gnfs doesn't lend
itself to adding NFSv4 and beyond.

NFS-Ganesha has NFSv3, NFSv4, NFSv4.1, NFSv4.2, pNFS — and if you care
about it, 9P support. It's a better implementation, and it's being
actively developed and maintained.

Besides kernel NFS (knfs) it doesn't make sense to pay for maintaining
two very different NFS implementations. (IOW Red Hat isn't going to pay
to maintain and develop two different implementations. Someone else may
step up and maintain gnfs.)

You can keep using gnfs, but eventually you should switch to NFS-Ganesha
because that's where resources are devoted — for fixing bugs and adding
features.

You *should* switch, but nobody is going to force you to switch. The
gnfs part will always be there and you'll always be able to use it.

-- 

Kaleb
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Reasons for recommending nfs-ganesha

[te-yamauchi]

Hi Jiffin

I understood that glusterfs nfs supports only bug fixes.
I appreciate your support and help so much.

--
Yamauchi

> 
> Hi,
> 
> 
> On 19/05/17 18:27, te-yamau...@usen.co.jp wrote:
> > I currently use version 3.10.2.
> > When nfs is enabled, the following warning is displayed.
> > Why is nfs-ganesha recommended?
> > Is there something wrong with gluster nfs?
> >
> > Gluster NFS is being deprecated in favor of NFS-Ganesha Enter "yes" to
> > continue using Gluster NFS (y/n)
> 
> The main reason behind above warning message is that currently most of the
> development focus happens in NFS-Ganesha than gluster nfs(only bug fixes).
> The following are major plus points for NFS-Ganesha
> 1.) NFS-Ganesha is a different community which supports a lot of other
> fIlesystem like CEPH(cephFS / RGW), GPFS, Lustre
> 2.) It support different nfs protocols including v3,v4, v4.1 and pNFS where as
> gluster NFS supports only v3
> 3.) Can do dynamic addition/modification of exports(shares) , where in gluster
> nfs each time server requires restart 4.)It has an integrated HA solution 
> using
> pacemaker & corosync for gluster volumes
> 
> --
> Jiffin
> 
> 
> > ___
> > Gluster-users mailing list
> > Gluster-users@gluster.org
> > http://lists.gluster.org/mailman/listinfo/gluster-users

___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Script to monitor cpu/memory utilization of processes

[Arthy Loganathan]

Hi All,

I have come up with a python script[1] to monitor cpu, memory 
utilization of the processes and load average of the system in regular 
intervals of time.
It collects the data in an excel sheet for later reference. It draws 
chart with the monitored system data on demand. Please refer link [2] 
for the help guide.


[1] https://github.com/aloganat/system_monitor

[2] https://github.com/aloganat/system_monitor/blob/master/README.md

Please give it a try. Do provide your feedback/suggestions on how it can 
be customized/tailored to assist your projects further.



Thanks & Regards,
Arthy


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] URGENT - Cheat on quorum

[Ravishankar N]

On 05/22/2017 11:02 AM, WK wrote:




On 5/21/2017 7:00 PM, Ravishankar N wrote:

On 05/22/2017 03:11 AM, W Kern wrote:


gluster volume set VOL cluster.quorum-type none

from the remaining 'working' node1 and it simply responds with

"volume set: failed: Quorum not met. Volume operation not allowed"

how do you FORCE gluster to ignore the quorum in such a situation?

You probably also have server quorum enabled 
(cluster.server-quorum-type = server). Server quorum enforcement does 
not allow modifying volume options or other actions like peer 
probing/detaching if server quorum is not met.




Great, that worked.  ie  gluster volume set VOL 
cluster.server-quorum-type none


Although I did get an error  of "Volume set: failed: Commit failed on 
localhost, please check the log files for more details"


but then I noticed that volume immediately came back up and I was able 
to mount the single remaining node and access those files.


So you need to do both settings in my scenario.


Also, don't disable client quorum for arbiter volumes or you will end 
up corrupting the files. For instance, if the arbiter brick was the 
only one that is up, and you disabled client quorum, then a writev 
from the application will get a success but nothing will ever get 
written on-disk on the arbiter brick.


Yes, I am learning the pro/cons of arbiter and understand their 
limitations.


In this test case, I had taken the arbiter OFFLINE (deliberately) and 
I was rehearsing a scenario where only one of the two real copies 
survived and I needed that data.  Hopefully that is an unlikely 
scenario and we would have backups, but I've earned the grey specs in 
my hair and the Original Poster who started this thread run into that 
exact scenario.


On our older Gluster installs without sharding, the files are simply 
sitting there on the disk if you need them. That is enormously 
comforting and means you can be a little less paranoid compared to 
other distributed storage schemes we use/have used.


But then I have my next question:

Is it possible to recreate a large file (such as a VM image) from the 
raw shards outside of the Gluster environment if you only had the raw 
brick or volume data?


From the docs, I see you can identify the shards by the GFID

# getfattr -d -m. -e hex/path_to_file/
# ls /bricks/*/.shard -lh | grep /GFID

Is there a gluster tool/script that will recreate the file?

or can you just sort them sort them properly and then simply cat/copy+ 
them back together?


cat shardGFID.1 .. shardGFID.X > thefile
/


Yes, this should work, but you would need to include the base file (the 
0th shard, if you will) first in the list of files that you're stitching 
up.  In the happy case, you can test it by comparing the md5sum of the 
file from the mount to that of your stitched file.

-Ravi

/
Thank You though, the sharding should be a big win.

-bill/
//



___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Couple of questions

[Amar Tumballi]

Hi Chris,

Some answers inline.

On Sun, May 21, 2017 at 2:17 AM, Chris Knipe  wrote:

> Hi All,
>
>
>
> I have a couple of questions which I hope that there’s someone to just
> shed some light on for me please.  I don’t think it’s anything serious,
> other than really just trying to better understand how the underlying
> GlusterFS works.
>
>
>
> Firstly, I plan to build a GluserFS using SM 5018D8-* boxes.  Essentially,
> 12 x 10TB disks, 128GB Ram, and a Xeon D-1537 CPU.  It does have an
> on-board LSI RAID controller, but there’s not a lot of detail forthcoming
> in terms of RAID configurations (caches for example).
>
>
>
> Firstly, in terms of the nodes, I don’t care TOO much for data integrity
> (i.e. it is OK to lose SOME of data, but availability in terms of
> underlying hardware is more important).  Secondly, it may not be the
> perfect scenario for GlusterFS (although it works perfectly fine currently
> through NFS on standard servers), but we are talking about millions of >
> 500K < 1M files.  Files are stored in a very specific structure, so each
> file is read/written precisely to a unique directory.  There’s no expensive
> scanning of directories (i.e. ls) happening or anything like that.  It’s a
> simple and very static read/write operation for each file on the system.
>
>
>
> Currently we store articles using a MD5 hash algorithm for a file name,
> and use 5 directory levels, so, /a/a0/a02/a02b/a02ba/
> a02ba1234567813dfa23bd2348901d33 Again everything works fine using
> multiple servers and standard ext4 / nfs exports.  We host /a on one
> server, /b on another server, etc.  So whilst the directories (and IO load)
> is split to address load issues, we are a bit limited in terms of how and
> how much we can expand.  I’m hoping to move all of this to GlusterFS.  The
> applications are very random IO intensive, and whilst we are nowhere CLOSE
> to the capabilities of the hardware, it is actually the seek times that are
> our limiting factor and the biggest bottleneck.  Therefore, I am fairly
> certain that growing through NFS, or, GlusterFS should be suitable and
> workable for us.
>
>
>
> My main reason for wanting to go GlusterFS is mostly related to better and
> easier expansion of storage.  It seems that it is easier to manage, whilst
> also providing some degree of redundancy (even if only partially in the
> case of a Distributed volume, which I believe would be adequate for us).
> All drives are hot swappable, and we will more than likely either look at a
> Distributed, or Stripped volume.  In the case of a Distributed system, we
> can still live with the fact that the majority of files remain available,
> whilst a certain amount of files becomes unavailable should a node or brick
> fail, so Distributed will more than likely be adequate for our needs.
> Stripped would be nice to have, but I think it would have some complexities
> given our specific use case.  We are also talking high concurrently (we do
> about 6K read/writes per second over NFS currently, per NFS server)
>
>
>
> 1 On the client(s), mounting the GlusterFS the documentation is clear in
> that it will only fetch the GlusterFS configuration, whilst there after
> reading/writing directly to the GlusterFS nodes.  How non-stop is this?  If
> there is already a mount made and additional nodes are added / removed from
> the GlusterFS, does the client(s) get informed of this without the need to
> re-mount the file system?  What about the capacity on the mount (at the
> client) when a node is added?  Basically, how non-stop is this operation?
> Can I assume that (in a perfect world) the client would never need to
> re-mount the file system? Are there any operations in GlusterFS that would
> require a client to re-mount?
>

No need to re-mount. GlusterFS fetches the volume config changes from the
server from which it is mounted, and gives the scaled out storage layout to
its clients / applications.


>
>
> 2 Given the Distributed nature of GlusterFS and how files are written to
> nodes, would it be safe to assume that how more nodes there are in the
> GlusterFS, how better the IO performance would be?  Surely, the IO load is
> distributed between the nodes, together with the individual files, right?
> What kind of IO could (or should) reasonably be expected given the hardware
> mentioned above (yes, I know this is a how long is a piece of string
> question)?
>

Here, the performance improvements can be seen when there are more clients
using the volume while you add the server. Most of the cases when you keep
the number of clients same, client n/w would become bottleneck to not see
any performance impact. But if you increase the client too along with
server, in general we see linear improvement in performance for file I/O.


>
>
> 3 When bricks are added / removed / expanded / rebalanced / etc… What does
> GlusterFS actually do?  What would happen for instance, if I have a 250TB
> volume, with 10M files on it, and I add another node